LLM Research Papers

LLM Research Papers

ICML 2025 Papers

ICML 2025 Papers

Game Theory in AI and Computer Science

Game theory, a mathematical framework for analyzing strategic interactions among rational decision-makers, has become an indispensable tool in computer science and artificial intelligence. It provides a powerful lens through which to model complex systems where multiple autonomous agents, whether human or AI, pursue their own objectives, often with interdependent outcomes. Understanding how to apply and codify game theory algorithms is crucial for designing intelligent systems that can navigate competitive and cooperative environments effectively.

At its heart, applying game theory involves defining the "game": identifying the players (agents), their available actions or strategies, the payoffs (utility or reward) associated with each combination of strategies, and the information available to each player. Once formalized, various algorithms can be employed to predict outcomes or prescribe optimal strategies. Key concepts include the Nash Equilibrium, a stable state where no player can improve their outcome by unilaterally changing their strategy, assuming others keep theirs constant. Algorithms like Support Enumeration or the Lemke-Howson algorithm can compute Nash Equilibria in certain game types, though exact solutions for complex games can be computationally challenging. For simpler, perfect-information, zero-sum games (where one player's gain is another's loss), the Minimax algorithm is often used, enabling an AI to choose moves that minimize the maximum possible loss, assuming an optimal opponent.

The application areas are vast. In multi-agent systems (MAS), game theory helps design interactions between autonomous agents, enabling them to coordinate tasks, allocate resources, and even negotiate. For instance, in a swarm of drones, game theory can optimize flight paths to avoid collisions or collaboratively search an area. In cybersecurity, it models the adversarial relationship between attackers and defenders, allowing the development of robust defense strategies such as randomized patrols or adaptive threat responses. Resource allocation in cloud computing, network routing, and even dynamic pricing strategies in online markets are other significant domains where game theory algorithms inform optimal decision-making.

Codifying these algorithms in computer science and AI applications involves several steps. Firstly, accurately representing the game state and player information is paramount, often using data structures like matrices for payoffs or trees for sequential games. Secondly, defining utility functions that precisely quantify the objectives of each agent is crucial, as these drive the decision-making process. Algorithms then operate on these representations. For example, implementing Minimax often involves a recursive function that explores future game states, assigning scores based on predicted outcomes. For Nash Equilibrium, codification might involve iterative methods like Fictitious Play or Regret Matching, where agents learn optimal strategies over time by observing opponents' past actions and adjusting their own. More advanced AI techniques, such as Reinforcement Learning, can be deeply integrated with game theory, allowing agents to learn optimal strategies through trial and error in complex, dynamic game environments, often converging towards game-theoretic equilibria without explicit programming of the strategies.

Challenges exist, particularly in games with incomplete information, numerous players, or continuous strategy spaces, which can lead to computational intractability. However, the ongoing research in algorithmic game theory, combining insights from economics, mathematics, and computer science, continues to push the boundaries, enabling increasingly sophisticated and strategic AI behaviors in real-world applications.

How to make drones smarter

Drones have revolutionized various industries, from logistics to surveillance, yet their full potential remains untapped. The next frontier lies in imbuing these aerial vehicles with advanced intelligence, enabling them to operate with unprecedented autonomy, resilience, and efficiency. This evolution demands a multi-faceted approach, integrating concepts of self-awareness, self-correction, self-reactivity, self-adaptation, collective intelligence, and robust fault tolerance.

At the core of a truly intelligent drone is self-awareness. This extends beyond simple sensor readings; it involves the drone building and maintaining a comprehensive internal model of its own state (battery level, component health, flight dynamics) and its dynamic environment (weather conditions, air traffic, obstacle maps). Through sophisticated AI algorithms, a self-aware drone can interpret raw sensor data, understand its current mission context, and even predict potential future states, forming the bedrock for intelligent decision-making.

Building upon self-awareness, self-correction and self-reactivity enable a drone to respond dynamically to unforeseen circumstances. A self-correcting drone can detect deviations from its planned trajectory or performance metrics and automatically adjust its controls to maintain stability and mission objectives. Self-reactivity, on the other hand, allows for immediate, intelligent responses to sudden external events, such as a rogue bird, an unexpected gust of wind, or a sudden system malfunction. This involves rapid re-computation and execution of new flight paths or operational adjustments in real-time.

Self-adaptation takes intelligence a step further, allowing drones to learn and evolve their behavior over time. Through machine learning and reinforcement learning techniques, a self-adaptive drone can analyze past performance, identify optimal strategies for different scenarios, and refine its internal models. This enables it to improve its efficiency, navigation, and task execution with every flight, adapting to new terrains, changing mission parameters, or even evolving environmental conditions.

The true power of autonomous drones will be realized through collective intelligence. Imagine a swarm of drones communicating seamlessly, sharing sensor data, and distributing tasks based on real-time needs and individual capabilities. This collective brain allows for shared situational awareness, collaborative problem-solving, and distributed decision-making, making the entire system more robust and capable than any single drone. Tasks too complex for one drone can be tackled by a coordinated fleet, optimizing resource allocation and mission success.

Finally, fault tolerance is paramount for reliable autonomous operation. This involves designing drones with redundant systems, both hardware and software, to ensure graceful degradation rather than catastrophic failure. Intelligent fault detection mechanisms can identify anomalies, isolate failing components, and reconfigure the system to continue operating, albeit perhaps at a reduced capacity. Self-healing capabilities, where a drone can autonomously repair or compensate for minor damage, further enhance resilience, ensuring missions are completed even in challenging conditions.

The journey towards truly smart and self-aware drones is an exciting one, driven by advancements in artificial intelligence, sensor technology, and robust system design. By integrating self-awareness, self-correction, self-reactivity, self-adaptation, collective intelligence, and fault tolerance, we are moving beyond remotely piloted aircraft towards autonomous, intelligent agents capable of complex operations, promising a future where drones work seamlessly and safely alongside humans in an ever-expanding array of applications.

Game Theory and Actor-Critic in Peer Review

Peer review is a cornerstone of academic and professional quality control, designed to ensure the rigor, validity, and integrity of published work. However, the process itself is not without its challenges. Issues such as reviewer bias, inconsistent feedback, and the free-rider problem (where some reviewers contribute less than others) can undermine its effectiveness. Integrating concepts from game theory and the actor-critic reinforcement learning approach offers a sophisticated framework to model, analyze, and potentially optimize the peer review system, fostering more equitable and efficient outcomes.

Game theory provides a powerful lens through which to view peer review as a strategic interaction among rational agents. Each participant—authors, reviewers, and editors—has specific objectives and makes decisions based on their perceived payoffs. For instance, authors aim for publication and constructive feedback, reviewers seek recognition or intellectual engagement, and editors strive for high-quality, timely reviews. The "game" involves reviewers deciding on the effort they exert, the honesty of their critique, and their timeliness, while authors might strategize on where to submit their work and how to respond to feedback. By understanding the incentives and potential Nash equilibria (stable states where no player can improve their outcome by unilaterally changing their strategy), we can identify systemic weaknesses and design mechanisms that encourage more desirable behaviors. For example, a game-theoretic analysis might reveal that without proper incentives or accountability, reviewers might choose to exert minimal effort, leading to superficial reviews.

Building upon this, the actor-critic approach from reinforcement learning can be applied to dynamically improve the peer review process. In this context, the "actor" is a policy that decides on actions (e.g., how to assign papers to reviewers, how to incentivize reviewers, or how to aggregate diverse feedback), while the "critic" evaluates the quality of these actions. The environment is the peer review system itself, and rewards could be tied to metrics like review quality, consistency, timeliness, and author satisfaction.

Imagine an actor-critic system operating within a peer review platform. The actor might learn to assign papers to reviewers based on their past performance, expertise, and current workload, aiming to optimize for review quality and turnaround time. The critic would then assess the outcome of these assignments—did the reviews meet quality standards? Was the paper published successfully? Did the authors find the feedback useful? Based on the critic's evaluation, the actor's policy is updated, allowing the system to continuously learn and refine its strategies. This iterative feedback loop helps the system discover optimal assignment policies, identify effective incentive structures, and even detect potential biases, leading to a more robust and fair review process.

The integration of game theory and the actor-critic approach holds immense promise for transforming peer review. Game theory helps us understand the underlying strategic dynamics and potential pitfalls, while the actor-critic model provides a practical, adaptive mechanism for real-time optimization. Such a system could lead to more efficient allocation of reviewing resources, higher quality and more consistent feedback, and ultimately, a more reliable and respected scholarly communication ecosystem. While implementation would require careful design of reward functions and robust data collection, the potential benefits for advancing knowledge and ensuring quality are substantial.

Multi-Armed Bandits in Search Engine Marketing

In the dynamic world of Search Engine Marketing (SEM), advertisers constantly grapple with the challenge of allocating budgets and optimizing campaigns for maximum return on investment. Traditional A/B testing, while valuable, can be slow and inefficient, especially when dealing with a multitude of ad creatives, keywords, or bidding strategies. This is where the concept of Multi-Armed Bandits (MAB) offers a powerful and agile alternative, enabling real-time optimization and significantly improving campaign performance.

A Multi-Armed Bandit problem is a classic scenario in reinforcement learning where an agent must choose between multiple options (the "arms" of a slot machine) to maximize its cumulative reward over time. Each arm has an unknown probability distribution of rewards, and the agent must balance "exploration" (trying new arms to discover their true potential) with "exploitation" (sticking with the arms that have historically yielded the best rewards). In SEM, each "arm" can represent a different ad creative, a specific keyword bid, a landing page variant, or even a distinct audience segment. The "reward" is typically a conversion, a click-through rate, or a specific cost-per-acquisition target.

Consider a scenario where an advertiser has five different ad headlines for a single product. Instead of running a lengthy A/B test where traffic is split evenly, a MAB algorithm can dynamically adjust traffic distribution. Initially, it might send a small amount of traffic to each headline (exploration). As data comes in, the algorithm identifies which headlines are performing better (e.g., higher click-through rates or conversion rates). It then gradually allocates more traffic to the better-performing headlines (exploitation), while still sending a small percentage to the less effective ones to ensure it doesn't miss out on a headline that might improve over time or under different conditions. This continuous learning and adaptation allow for faster identification of winning strategies and a more efficient use of advertising spend.

The advantages of applying MAB to SEM are considerable. Firstly, it enables faster optimization. Unlike traditional A/B testing that requires a fixed duration or sample size, MAB algorithms can adapt in real-time, quickly shifting resources to the most effective elements. Secondly, it leads to higher efficiency and ROI. By continuously directing traffic to better-performing variations, MAB minimizes wasted ad spend on underperforming elements, thereby maximizing the return on advertising investment. Thirdly, MAB inherently handles the exploration-exploitation trade-off, ensuring that campaigns not only leverage what's currently working but also discover new, potentially superior strategies. This is particularly crucial in fast-changing markets where consumer behavior or competitive landscapes can shift rapidly.

While MAB offers significant benefits, its implementation in SEM requires careful consideration. Defining clear, measurable rewards (e.g., conversions, revenue) is paramount. The choice of MAB algorithm (e.g., Epsilon-Greedy, Upper Confidence Bound, Thompson Sampling) depends on the specific campaign goals and data characteristics. Furthermore, integrating MAB systems with existing ad platforms and ensuring data privacy and security are practical challenges that need to be addressed.

Multi-Armed Bandits provide a sophisticated and highly effective framework for optimizing SEM campaigns. By enabling continuous, adaptive learning and real-time resource allocation, MAB algorithms empower advertisers to navigate the complexities of online advertising with greater agility and precision, ultimately driving superior performance and maximizing the value of every advertising dollar. 

Reinforcement Learning in Mobile Navigation

The ubiquitous nature of mobile devices has transformed how we interact with the internet. Yet, despite advancements in web design, navigating complex mobile websites can still be a frustrating experience. Small screens, often cluttered layouts, and the inherent imprecision of touch interfaces contribute to a less-than-ideal user journey. This is where reinforcement learning (RL) offers a compelling solution, promising to revolutionize mobile web navigation by making it more intuitive, personalized, and efficient.

Reinforcement learning, a paradigm of machine learning, involves an "agent" learning to make optimal decisions by interacting with an "environment." The agent performs "actions" and receives "rewards" or "penalties" based on the outcome, iteratively refining its strategy to maximize cumulative reward. In the context of mobile web navigation, the website itself is the environment, the user (or an AI proxy) is the agent, and clicking or swiping are the actions. A positive reward could be reaching a desired page quickly, completing a purchase, or spending a significant amount of time on relevant content, while a negative reward might be a bounce, a dead-end, or excessive scrolling.

By applying this framework, an RL agent can learn individual user preferences and common navigation patterns. Imagine a user frequently visiting a specific product category on an e-commerce site. An RL system could observe this behavior, associate it with positive rewards (e.g., adding items to a cart), and then dynamically adjust the website's interface. This might involve prominently displaying links to that category, prioritizing search results, or even suggesting a direct path to frequently accessed sections. The goal is to anticipate the user's next likely action, minimizing the cognitive load and physical taps required to achieve their objective.

The benefits of such an approach are multifaceted. For users, it translates to a significantly improved experience: less time spent searching, fewer mis-taps, and a more seamless flow through content. This heightened efficiency reduces frustration and allows users to accomplish tasks faster, whether it's finding information, making a booking, or engaging with multimedia. For website owners, this directly correlates to increased user engagement, higher conversion rates, and reduced bounce rates, ultimately leading to better business outcomes. Furthermore, an RL-driven navigation system could enhance accessibility by adapting to different interaction styles or even predicting the needs of users with specific impairments.

However, implementing reinforcement learning for mobile navigation is not without its challenges. One primary hurdle is the need for vast amounts of high-quality user interaction data to train the RL models effectively. Designing an appropriate reward function is also critical; it must accurately reflect successful user journeys and avoid incentivizing undesirable behaviors. There's also the classic exploration-exploitation dilemma: how much should the system try new navigation strategies (exploration) versus sticking to proven successful ones (exploitation)? Computational overhead on mobile devices and concerns regarding user data privacy also require careful consideration and robust solutions.

Despite these complexities, the potential of reinforcement learning to transform mobile web navigation is immense. As RL algorithms become more sophisticated and computational resources more accessible, we can anticipate a future where mobile websites are not just responsive, but truly intuitive – learning from our interactions to offer a personalized and effortlessly efficient browsing experience. This shift promises to make the mobile web a far more enjoyable and productive space for everyone.

Proximal Policy Optimization

Proximal Policy Optimization

Proximal Policy Optimization

Proximal Policy Optimization Algorithms

Distributional Reinforcement Learning

Distributional Reinforcement Learning

Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning

Graph-Based Reinforcement Learning

• Reinforcement Learning on Graphs
• Playing Text-Adventure Games
• Graph-Based Reinforcement Learning for Active Learning in Real Time
• Graph-Based Reinforcement Learning Method with Converged State Exploration and Exploitation

Micromouse

 

Lenskit

Lenskit

Drawbacks of Reinforcement Learning

• Reproducibility
• Resource Efficiency
• Susceptibility to Attacks
• Explainability/Accountability

Reinforcement Learning Cheatsheet

Swarm Bandit Robotics

Border control is an issue for most land locked countries including ones that have direct access to water ways. Being able to control every aspect of a border is not humanly possible in most cases. But AI can in fact help cover a larger distance with greater amount of force and attributed control. Artificial Intelligence in form of swarm intelligence and reinforcement learning can create an effective force for border security and control. Building a great wall is pointless and overly expensive. Eventually the wall comes down. But, an army of drones and robots, if well engineered, becomes a force to reckon with as well as adaptable to patterns of attack and infiltration. The ultimate goal being an autonomous army of swarm robots that can apply tactical understanding and manoeuvrability across the entire map of the country, advanced in strategic alliance and combat, when necessary, to protect the sovereignty of a nation and its people.

Military Swarming
Kilobots
pentagon drone swarm autonomous war machines

Gym Reinforcement Learning From OpenAI

Gym
OpenAI on Github