CCNets

CCNets: Technical Overview We want to introduce our new machine learning algorithm, Causal Learning, which integrates supervised and unsupervised learning into a unified framework. Like supervised learning, this algorithm is designed to work across various data types, including tabular, images, and time-series data. Our approach leverages three neural networks—explainer, reasoner, and producer—that learn to make predictions and generate data by understanding causal relationships between X and Y in datasets. Our model training operates differently from numerous other methods based on supervised or unsupervised (generative) learning. It allows us to offer distinct advantages over current machine learning approaches. We hope many users will benefit from this innovative addition to their machine learning toolkit.

1-Click Robotics: Final Testing in Progress! We are excited to announce that our 1-Click Robotics remote AI agent training service is now in the final stages of testing. Soon, we’ll release an open-source repository that integrates seamlessly with the Gymnasium interface, allowing you to easily connect your own environments. This service acts as a gateway, facilitating the transfer of environment states and actions for efficient AI training. After training, whether your environment runs locally or on AWS, our decision models are designed to effectively meet your specific needs, whether in gaming, drone missions, or beyond. From Unity to Unreal Engine and other platforms, our trained models ensure smooth and reliable performance for your AI agents. Stay tuned for the official release!

1-Click AI Agent Training and Performance Control Service via AWS As we prepare to launch our 1-click AI agent training service on AWS, we want to share performance results and highlight new controllability features. In the above HalfCheetah-v5 game, our model(onnx) showed dynamic performance control over 100 episodes. By varying an external control value from 100% down to 0% and back to 100%, we modulated the agent's score from 8,000 points down to nearly zero and back up again. The accompanying graph shows how the agent's performance closely tracks the control value, highlighting the model's ability to precisely regulate performance in real-time. Why Is This Important? By accepting both the agent's state and an external control value, our GPT Decision Control Model enables precise management of agent behavior to limit performance or achieve specific target scores by the end of the game. This provides precise control over training outcomes across diverse game environments. We look forward to helping you explore new possibilities for AI-enhanced gaming solutions. Check out our progress here: https://lnkd.in/gFq_b_yd

Game Balance Control AI Agent Training on AWS We’re excited to announce a new milestone: achieving an impressive 9500 episode score(https://lnkd.in/gFq_b_yd) on Humanoid-v4 Gymnasium (current benchmark under 7000: https://lnkd.in/gS7Gimiq) with our Causal Reinforcement Learning algorithm! This high-performing algorithm will soon be available on AWS SageMaker. Our service enables you to train Causal AI agents with minimal setup, delivering the same highest level of performance. A detailed guide on Marketplace interaction is on the way, simplifying the training service to support your success in game deployment. Highlights: - Game Environment Wrapping with Gymnasium Interface: Seamlessly integrate your game state for real-time training feedback, while fully protecting your game’s IP. - From 2D Mobile to 3D AAA Games: Uniform GPT-2-based models in a consistent, generic size adapt seamlessly to games of all levels—from simple to advanced play (across from Hopper to Humanoid). - Tuning-Free with Hyperparameter Flexibility: Start training instantly, with optional hyperparameter tuning available for enhanced performance.

GPT-2 Agent Benchmark Score with 1-Click Solution We are excited to announce that we have achieved best-in-class performance using a one-click, uniform setting targeting 3D physics environments like MuJoCo (PaperswithCode sota: https://lnkd.in/gS7Gimiq). Small GPT models (GPT-2) are used to attain top scores in these 3D environments. Check out our progress here: https://lnkd.in/gFq_b_yd Key Highlights: 1. 5x Fewer Steps: Significantly reduced the number of simulation steps, cutting down gameplay costs. Reducing steps is crucial since many updates must be performed sequentially and cannot be parallelized. 2. No Tuning Required: From simple one-leg Hopper to complex Humanoid agents, training parameters (like gamma and lambda) are dynamically learned. We aim for no tuning across general scales of games. 3. Unified Model Configuration: Utilized uniform GPT-2 models with the same configuration across all tasks to save time during testing and deployment. While our SDK is on the way, we invite you to explore our algorithm and the details of how it works: https://lnkd.in/gXzT8QpR What's Next: - Preparing benchmark scores on 2D games such as Atari by adding an extra image encoder in this 1-click setting. - Testing results with different computational scales in cloud services and planning to offer this high-performing AI agent modeling as a cloud service soon.

Coming Soon: SDK for Building Revenue-Maximizing Apps with GPT Decision Models Trained on Our Cloud Service Benefits: 1. Test Before You Train: Try our sample models (trained by us) on OpenAI Gym environments via WandB to validate top-tier performance benchmarks—no upfront cloud costs. 2. Direct Control of GPT Decision Models: Use our SDK to manage, test, and deploy GPT models directly in your environment, giving you full control over their behavior. This saves time, reduces costs, and ensures optimal app performance. 3. End-Game Score Control Capability: Our Causal RL algorithm trains GPT models to utilize the current state and "target end score." The model predicts actions to ensure the game ends with the desired score, giving you full control over game outcomes for maximum revenue potential. Test this precise gameplay control to create engaging, dynamic scenarios that boost player interaction and drive monetization. Check out our project workspace: https://lnkd.in/g-4KA6VD Algorithm details: https://lnkd.in/gXzT8QpR

New robotic algorithm for training GPT for AI agents is ready to launch on the cloud. DeepMind introduced a top RL model based on RNNs (Agent57), but why has OpenAI chosen GPT models instead of RNNs in language generation? The key likely lies in the attention mechanism within GPT models, which excels at handling general and diverse user environments—something RNNs struggle with due to their inherent limitations. At CCNets, we have enabled the training of GPT models in the RL domain using a uniform setting for general environments (Mujoco, Atari, ML-Agents)—similar to how ChatGPT generates language for everyone today without even knowing the specifics of clients' environments. Our RL algorithm, Causal RL, optimizes both value and policy to achieve the best results. Additionally, we’ve integrated a reverse-environment network, allowing agents to “rewind” and learn from actions that minimize future value by undoing them. We train GPT for both policy and value by leveraging the reverse-environment network. If this algorithm can train GPT models for more general environments, could we soon envision robots walking among us—an extension of what ChatGPT does in the real world? Check out our project workspace: https://lnkd.in/g-4KA6VD Algorithm details: https://lnkd.in/gXzT8QpR

What's After GenAI? GenAI adds a lot of value to our work, but there are more important problems that need actions in the real world rather than just text generation. Reinforcement learning (RL) trains agents to take meaningful actions in complex environments, directly addressing these real-world needs. To train agents that can function in general environments, RL must determine when training should end (i.e., when no further improvement is possible in the environment) and ensure that agents will do the "right" actions that meet our needs. But how can we trust that they will do what we want? How can we be sure that the agent has learned the best it can from training? At CCNets, our Causal RL algorithm builds on traditional RL by optimizing both value and policy to maximize outcomes. Additionally, it includes a reverse-environment network. Our training system doesn’t just reverse an agent's state at termination; it rewinds every action that changes the agent's state, learning from actions that also "minimize" future value by undoing the action. The more the reverse-environment minimizes future value, the more the policy network is induced to maximize rewards. This approach re-simulates state transitions both forward and backward in the buffer iteratively. With bidirectional safety checks that consider the consequences of actions, similar to time travel, could we envision robots walking among us soon? Check out our project workspace: https://lnkd.in/gN2a23j2

Next Generation RL with Causal Learning. Our 1-Click cloud service is built on: 1. Actor-Critic Framework: Provides foundational RL components for policy and value estimation. 2. Reverse-Environment Network: Integrates a GPT model to learn the reverse dynamics of the environment, enabling agents to use invertible reasoning in their decision-making processes. 3. GPT Reinforcement Learning: Utilizes GPT-2 models for the Actor, Critic, and Reverse Environment networks, fostering the learning and generation of context-aware actions. 4. All Setting Auto-tuning: Dynamically adjusts parameters such as gamma, lambda, reward scale, and learning rate by leveraging the mechanisms of the CCNets framework. For more information, please visit our newly launched website at CCNets.org. Learn how our algorithm works: https://lnkd.in/gXzT8QpR

What if GPT can learn to operate actions instead of just generating answers? What if it could simulate time transitions between the present and the future, while considering the consequences of its actions using a reverse environment? Imagine how many problems it could solve. CCNets introduce Causal Reinforcement Learning that trains three GPTs simultaneously: the Action Operator GPT, the Value Estimator GPT, and the Reverse-environment Simulator GPT. We are nearing the completion of our Reinforcement Learning GPTs solution, which is designed to be completely tuning-free. We aim to deliver it soon via our Cloud Service. For OpenAI Gym benchmark results, visit our: https://lnkd.in/gN2a23j2