GPT-4o System Card
Pith reviewed 2026-05-23 18:38 UTC · model grok-4.3
The pith
GPT-4o is an autoregressive omni model that accepts any combination of text, audio, image, and video inputs and generates any combination of text, audio, and image outputs through a single end-to-end neural network.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It is trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50% cheaper in the API, and shows particular gains in vision and audio.
What carries the argument
The single end-to-end autoregressive neural network that unifies processing and generation across text, vision, and audio modalities.
If this is right
- The model enables faster and lower-cost API use while retaining or exceeding prior text performance.
- It delivers stronger results on non-English text, vision tasks, and audio understanding than previous models.
- Speech-to-speech interactions become feasible with near-human latency.
- Third-party assessments of dangerous capabilities supplement the internal safety evaluations.
Where Pith is reading between the lines
- Widespread adoption could shift how people interact with AI in everyday settings that mix speech and images.
- The unified architecture might reduce the need for separate specialized models in applications like translation or visual question answering.
- The reported societal impacts section implies that scaling such models will require ongoing attention to new use cases beyond the evaluated categories.
Load-bearing premise
The safety evaluations and alignment measures described are sufficient to address the risks posed by the model's capabilities.
What would settle it
A measurement showing average audio response times well above 320 milliseconds or performance below GPT-4 Turbo levels on English text and code tasks would challenge the stated claims.
Figures
read the original abstract
GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript is the GPT-4o System Card. It claims that GPT-4o is an autoregressive omni model accepting any combination of text, audio, image, and video inputs and generating any combination of text, audio, and image outputs; that it is trained end-to-end across modalities; that it responds to audio in as little as 232 ms (average 320 ms); that it matches GPT-4 Turbo on English text and code while improving on non-English languages, vision, and audio; that it is faster and 50% cheaper via API; and that it has undergone safety evaluations, Preparedness Framework assessments, third-party reviews on dangerous capabilities, and discussion of societal impacts, consistent with voluntary White House commitments.
Significance. If the reported metrics and evaluations are accurate, the document supplies useful public information on the capabilities and risk profile of a frontier multimodal model, supporting transparency and community awareness. The inclusion of third-party assessments on dangerous capabilities is a constructive element that could aid independent scrutiny.
major comments (2)
- [Abstract] Abstract: the latency claims (minimum 232 ms, average 320 ms for audio response) are stated without any description of measurement methodology, number of trials, hardware configuration, statistical variability, or raw data, preventing verification or reproduction of these central performance assertions.
- [Safety evaluations paragraph] Safety evaluations paragraph: the claims regarding safety, alignment, and risk mitigation rest entirely on internal evaluations whose protocols, benchmarks, red-teaming procedures, and quantitative results are not detailed, which is load-bearing for assessing whether the described measures adequately address the model's stated capabilities.
minor comments (1)
- [Abstract] The abstract refers to 'our voluntary commitments to the White House' without specifying which commitments are addressed by the evaluations presented in the card.
Simulated Author's Rebuttal
We thank the referee for the review and for highlighting these points on the GPT-4o System Card. We respond to each major comment below.
read point-by-point responses
-
Referee: [Abstract] Abstract: the latency claims (minimum 232 ms, average 320 ms for audio response) are stated without any description of measurement methodology, number of trials, hardware configuration, statistical variability, or raw data, preventing verification or reproduction of these central performance assertions.
Authors: The reported latency figures reflect internal measurements on production infrastructure under conditions intended to approximate real-world conversational use. We are unable to release the full methodology, trial counts, hardware details, or raw data, as these constitute proprietary information about our serving stack and evaluation harness. This level of detail is not provided in comparable system cards or technical reports from other frontier labs. The numbers are offered as indicative performance characteristics rather than as a reproducible benchmark. revision: no
-
Referee: [Safety evaluations paragraph] Safety evaluations paragraph: the claims regarding safety, alignment, and risk mitigation rest entirely on internal evaluations whose protocols, benchmarks, red-teaming procedures, and quantitative results are not detailed, which is load-bearing for assessing whether the described measures adequately address the model's stated capabilities.
Authors: The system card summarizes the scope of internal safety work, references the Preparedness Framework, notes third-party dangerous-capability reviews, and describes high-level mitigations. Full protocols, red-teaming procedures, and granular quantitative results are withheld to avoid disclosing information that could aid adversarial attacks or model misuse. This approach is consistent with prior OpenAI system cards and with the voluntary commitments referenced in the document. The card is intended to convey the overall risk posture rather than to serve as a complete audit trail. revision: no
- Detailed latency measurement methodology, trial counts, hardware configuration, and raw data
- Detailed internal safety evaluation protocols, benchmarks, red-teaming procedures, and quantitative results
Circularity Check
No circularity detected
full rationale
This is a descriptive system card reporting measured capabilities, latencies, and safety evaluations of a deployed model. It contains no derivations, equations, fitted parameters, predictions, or mathematical claims whose validity could depend on self-referential steps. The central statements are direct factual reports of architecture and benchmark results, with no load-bearing chain that reduces to its own inputs by construction.
Axiom & Free-Parameter Ledger
Forward citations
Cited by 60 Pith papers
-
UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL
UniQL is a human-verified benchmark providing aligned natural language questions and dialect-specific SQL queries for 16 SQL systems to evaluate cross-dialect generalization.
-
Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?
Introduces the TVR active viewpoint-matching task and TVRBench indoor simulation benchmark, where foundation models start at low single-digit success rates but reach 51.4% after visual-action SFT and multi-turn GRPO p...
-
VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents
VideoFDB is a new benchmark and LM-as-judge framework for evaluating full-duplex audio-visual-to-audio-visual conversational agents on nonverbal dynamics from real video calls.
-
Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation
M³Att poisons medical multimodal RAG by pairing covert textual misinformation with query-agnostic visual perturbations that increase retrieval of the bad content, causing LLMs to generate clinically plausible but inco...
-
From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generation
MLLMs exhibit a Mirage effect by bypassing circuit diagrams in favor of header semantics for Verilog generation; VeriGround with identifier anonymization and D-ORPO training reaches 46% Functional Pass@1 while refusin...
-
SecGoal: A Benchmark for Extracting Formalizable Security Goals from Protocol Documents
The paper presents SecGoal, the first expert-annotated benchmark for security goal extraction from protocol documents, and demonstrates that fine-tuned 7B/9B parameter models achieve over 80% F1 score, outperforming l...
-
CHASM: Unveiling Covert Advertisements on Chinese Social Media
CHASM is a new benchmark dataset showing that existing multimodal large language models fail to reliably detect covert advertisements on Chinese social media even after fine-tuning.
-
HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models
HalluAudio is the first large-scale benchmark spanning speech, environmental sound, and music that uses human-verified QA pairs, adversarial prompts, and mixed-audio tests to measure hallucinations in large audio-lang...
-
EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformations
EVE enables verifiable self-evolution of MLLMs by using a Challenger-Solver architecture to generate dynamic executable visual transformations that produce VQA problems with absolute execution-verified ground truth.
-
HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?
Harmful skills in open agent ecosystems raise average harm scores from 0.27 to 0.76 across six LLMs by lowering refusal rates when tasks are presented via pre-installed skills.
-
ReConText3D: Replay-based Continual Text-to-3D Generation
ReConText3D is the first replay-memory framework for continual text-to-3D generation that prevents catastrophic forgetting on new textual categories while preserving quality on previously seen classes.
-
MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
MMRareBench provides 1,756 QA pairs and 7,958 images from PMC rare-disease cases to evaluate 23 MLLMs, revealing low treatment-planning scores and medical models underperforming general models on multi-image tasks due...
-
MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
MMRareBench is the first rare-disease benchmark for multimodal and multi-image clinical evaluation of MLLMs, revealing fragmented capabilities, low treatment-planning scores, and medical models underperforming general...
-
DialBGM: A Benchmark for Background Music Recommendation from Everyday Multi-Turn Dialogues
DialBGM is a new benchmark dataset revealing that existing AI models fall far short of human performance when recommending fitting background music for open-domain conversations.
-
EgoSound: Benchmarking Sound Understanding in Egocentric Videos
EgoSound is a new benchmark with 7315 QA pairs across seven tasks to evaluate egocentric sound understanding in multimodal large language models.
-
VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing
VLRS-Bench is the first benchmark dedicated to complex vision-language reasoning in remote sensing, with 2000 QA pairs across 14 tasks in cognition, decision, and prediction dimensions.
-
SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for Token-level Recognition of Semantic Differences Between Related Documents
SwissGov-RSD is the first naturalistic cross-lingual document-level benchmark with human token-level semantic difference annotations, on which both LLMs and encoders show a large performance gap relative to simpler settings.
-
MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
MVI-Bench supplies the first taxonomy and dataset focused on misleading visual inputs to measure LVLM robustness, with tests on 18 models revealing clear weaknesses.
-
Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
CritPt benchmark shows state-of-the-art LLMs reach only 5.7% average accuracy on full-scale unpublished physics research tasks, rising to about 10% with coding tools.
-
Flow-GRPO: Training Flow Matching Models via Online RL
Flow-GRPO is the first online RL method for flow matching models, raising GenEval accuracy from 63% to 95% and text-rendering accuracy from 59% to 92% with little reward hacking.
-
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo VLMs trained on newly collected PixMo open datasets achieve state-of-the-art performance among open-weight models and surpass multiple proprietary VLMs including Claude 3.5 Sonnet and Gemini 1.5 Pro.
-
LiveBench: A Challenging, Contamination-Limited LLM Benchmark
LiveBench is a contamination-limited LLM benchmark with auto-scored challenging tasks from recent sources across math, coding, reasoning and more, where top models score below 70%.
-
LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
LongEgoRefer is a new benchmark of 1,498 referring expressions in 45-minute average egocentric videos that exposes the failure of existing Video REC models on sparse long-form spatio-temporal grounding.
-
A Cost-Aware, Paired Protocol for Auditing Dynamic Tool Synthesis in Agentic Video Question Answering
Introduces a cost-aware paired protocol with six outcome groups and applies it to Dynamic-SAGE versus SAGE, reporting 7.5-point accuracy gain, 28% fewer tool calls, but 34% higher token use.
-
Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning
P2R decouples perception from reasoning in VLMs via a two-stage process and PRA-GRPO alternating RL training, reporting gains such as 93.2% on V-Star for the 4B model over its Qwen3-VL backbone.
-
EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes
EgoGapBench shows humans reliably select egocentric actions in multi-agent scenes while MLLMs systematically choose other agents' actions, and standard egocentric training data fails to close the gap.
-
(A)I Sees What You Don't: Exploiting New Attack Surfaces in Third-Party Mobile Agents
Identifies Screen Perception and Misused Channel attack surfaces in VLM-powered mobile agents and demonstrates seven attacks enabling arbitrary command execution on five frameworks without privileges.
-
SpheRoPE: Zero-Shot Optimization-Free 360 Panorama Generation with Spherical RoPE
SpheRoPE modifies rotary position embeddings in diffusion transformers to enforce spherical topology for zero-shot 360 panorama generation across multiple backbones.
-
No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs
Introduces VidPair-Halluc benchmark of 1K background-controlled adversarial video pairs and 11K QA pairs generated via PairFlow pipeline to evaluate hallucination in LVMs.
-
Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments
EQMs, sixty LLM-scored reasoning patterns, predict forecast accuracy at both item and person levels and outperform prior text-analysis methods in a large pre-registered tournament dataset.
-
OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning
OmniCoT is a new panoramic reasoning benchmark with 6.7K eval, 1K real, and 14.3K training examples plus a two-stage SFT+GRPO training method to enforce global 360-degree consistency.
-
MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs
MuseBench shows state-of-the-art MLLMs achieve only 48.29% accuracy on intent-level audiovisual arts understanding versus 87.18% for human experts.
-
A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents
A diagnostic framework called EPC reveals that proprietary LLM evaluators can exhibit large preference shifts between versions, as evidenced by a GPT-4o May-to-June drift that inverted study conclusions, rendering sin...
-
GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
GigaSpeechBench is a new 680-hour in-the-wild multilingual ASR/AST benchmark with five modules for low-resource languages, Chinese dialects, English accents, domain terminology, and age-varied speech, showing model pe...
-
HumanMoveVQA: Can Video MLLMs reason about human movement in videos?
HumanMoveVQA is a new benchmark that generates 10K+ QA pairs from 3D-lifted video tracks to evaluate video MLLMs on global human trajectory and orientation reasoning.
-
Reflect-R1: Evidence-Driven Reflection for Self-Correction in Long Video Understanding
Reflect-R1 introduces the first evidence-driven self-correction framework for long video understanding using a three-stage pipeline, stage-decoupled RL via SD-GRPO, and a 120K dataset to achieve SOTA on VideoMME and L...
-
Perception, Verdict, and Evolution: Hindsight-Driven Self-Refining Forensics Agent for AI-Generated Image Detection
ForeAgent combines a Perception-Verdict MLLM architecture with hindsight-driven self-refining via sampling-reflection-evolution to reach 82.18% accuracy on Chameleon and 93.3% mean accuracy across 16 generators on AIG...
-
PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing
PhyEditBench is a new benchmark for physics-aware image editing with real and synthetic instances plus a training-free PhyWorld baseline that uses test-time scaling to outperform SOTA models.
-
PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing
PhyEditBench is a new benchmark with real-world and synthetic instances that reveals limitations in current image editing models' physics reasoning and proposes a video-generation-based baseline called PhyWorld.
-
TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxy
TryOnCrafter is the first DiT-based framework for camera-controllable video virtual try-on via a renderable 4D try-on proxy distilled from 2D priors into 3DGS avatar animated with SMPL-X.
-
TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs
TriViewBench shows all tested MLLMs follow the same capability order with sharp drops on complex multi-view tasks and near-zero gain from Chain-of-Thought prompting.
-
CrypFormBench: Benchmarking Formal Analysis Capability of Large Language Models for Cryptographic Schemes
CrypFormBench is a new benchmark jointly covering symbolic and computational security to evaluate LLMs on five formal analysis capabilities, with results showing top model Claude-3.5 scores 48.7/100 and most models st...
-
Memory Retrieval in Visuomotor Policies for Long-Horizon Robot Control
HALO distills VLM priors via question-answering objectives and applies sparse attention to enable reliable memory retrieval from up to eight minutes of history in imitation-learned visuomotor policies.
-
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
ParaPairAudioBench is a new pairwise benchmark showing LALM judges lag human paralinguistic judgments by 32 percentage points with poor tie calibration across style, rate, emphasis, age, and gender.
-
NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Models
NegAS uses negative labels for attention guidance and sigmoid scoring to improve OOD detection in VLM-based object detectors while preserving ID performance.
-
Lost in Aggregation: A Multi-Scale Diagnostic Benchmark for LLM Spatial Navigation
A new diagnostic benchmark decomposes LLM spatial navigation into three cognitive scales and shows that cross-scale aggregation, not single-level deficits, causes failure beyond small mazes.
-
BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
BioMatrix unifies sequences, structures, and language for molecules and proteins inside one decoder-only foundation model via shared discrete tokens and achieves SOTA or competitive results on 77 of 80 downstream tasks.
-
GroundShot: Visually Consistent Multi-Shot Long Video Generation via Entity-Grounded Shot Scheduling
GroundShot introduces entity-grounded shot scheduling with online visual memory to improve consistency in multi-shot video generation and presents GroundBench for entity-level evaluation.
-
SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs
SPOT-E uses entropy shaping on answer predictions with low-entropy anchors to optimize visual spotlights at test time via GRPO for better VLM performance on evidence-intensive tasks.
-
ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models
ROSE benchmark shows MLLMs drop up to 44.5 percentage points from counting tasks to region-conditioned action on identical scenes, with the gap persisting even when counts are correct.
-
SIGMA: Skill-Incidence Graphs for Compositional Multi-Agent Design
SIGMA introduces skill-incidence graphs to compose agents from reusable skills, yielding higher average performance and robustness than topology-only baselines on reasoning and coding benchmarks.
-
Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients
ZPPO improves distillation to small vision-language models by using binary and negative candidate prompts plus a replay buffer for hard questions, outperforming standard distillation and GRPO on a 31-benchmark suite w...
-
Agentic AI Enhances Physician Trust in Clinical Decision Making
Empirical comparison shows physicians have higher cognitive and behavioral trust in agentic AI versus baselines on clinical cases, with noted over-reliance risk.
-
Multimodal Evaluator Preference Collapse: Cross-Modal Coupling in Self-Evolving Agents
Multimodal self-evaluation amplifies evaluator preference collapse and enables cross-modal contagion of strategy preferences, with self-evaluation showing near-complete immunity to the effect.
-
FARM: Find Anything using Relational Spatial Memory
FARM creates an open-vocabulary relational spatial memory that improves object retrieval recall by 164-224% over prior methods on 44k language queries across 67 scenes while running at 5-10 Hz.
-
FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation
FTP-1 is the first foundation tactile policy pretrained on ~3000 hours of data from 26 sources across 21 sensors that improves performance on seen setups by 17.2% and transfers to unseen sensors with 31% success rate gain.
-
AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages
AfriSUD supplies new SUD-annotated dependency treebanks for nine Sub-Saharan African languages and demonstrates that existing models exhibit clear limitations on their syntax.
-
Tail-Aware Adaptive-k: Query-Adaptive Context Selection for Retrieval-Augmented Generation
TAA-k finds query-adaptive retrieval cutoffs by first using knee detection to isolate a candidate window around the relevance-to-noise transition, then applying EVT goodness-of-fit tests inside that window.
-
Alignment Defends LLMs from Property Inference Attacks
Alignment defenses adapted from DPO and GRPO mitigate property inference attacks on LLMs while preserving utility.
-
From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
AVLLMs route audio-visual information sequentially in video tasks and via parallel streams for interleaved items, allowing early token discard with little performance loss across models and scales.
Reference graph
Works this paper leans on
- [1]
-
[2]
Universals and cultural variation in turn-taking in conversation,
T. Stivers, N. J. Enfield, P. Brown, C. Englert, M. Hayashi, T. Heinemann, G. Hoymann, F. Rossano, J. P. de Ruiter, K. E. Yoon, and S. C. Levinson, “Universals and cultural variation in turn-taking in conversation,”Proceedings of the National Academy of Sciences , vol. 106, no. 26, pp. 10587–10592, 2009
work page 2009
-
[3]
The White House, “Fact sheet: Biden-harris administration secures voluntary commitments from leading artificial intelligence companies to manage the risks posed by ai,” 2023
work page 2023
-
[4]
Openai preparedness framework beta,
OpenAI, “Openai preparedness framework beta,” 2023. https://cdn.openai.com/ openai-preparedness-framework-beta.pdf
work page 2023
- [5]
- [6]
-
[7]
OpenAI, “Gpt-4v(ision) system card.”https://openai.com/index/gpt-4v-system-card/, 2023. Accessed: 2024- 07-22
work page 2023
-
[8]
Navigating the challenges and opportunities of synthetic voices
OpenAI, “Navigating the challenges and opportunities of synthetic voices.” https://openai.com/index/ navigating-the-challenges-and-opportunities-of-synthetic-voices/ , 2024. Accessed: 2024-07-22
work page 2024
-
[9]
Warning: Humans cannot reliably detect speech deepfakes,
K. T. Mai, S. Bray, T. Davies, and L. D. Griffin, “Warning: Humans cannot reliably detect speech deepfakes,”PLoS One, vol. 18, p. e0285333, Aug. 2023
work page 2023
-
[10]
The uncanny valley [from the field],
M. Mori, K. F. MacDorman, and N. Kageki, “The uncanny valley [from the field],”IEEE Robotics & automation magazine, vol. 19, no. 2, pp. 98–100, 2012
work page 2012
-
[11]
How the voices for chatgpt were chosen,
OpenAI, “How the voices for chatgpt were chosen,” 2024
work page 2024
-
[12]
Evaluating the social impact of generative ai systems in systems and society,
I. Solaiman, Z. Talat, W. Agnew, L. Ahmad, D. Baker, S. L. Blodgett, C. Chen, H. D. I. au2, J. Dodge, I. Duan, E. Evans, F. Friedrich, A. Ghosh, U. Gohar, S. Hooker, Y. Jernite, R. Kalluri, A. Lusoli, A. Leidinger, M. Lin, X. Lin, S. Luccioni, J. Mickel, M. Mitchell, J. Newman, A. Ovalle, M.-T. Png, S. Singh, A. Strait, L. Struppek, and A. Subramonian, “E...
work page 2024
-
[13]
Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction,
R. Shelby, S. Rismani, K. Henne, A. Moon, N. Rostamzadeh, P. Nicholas, N. Yilla, J. Gallegos, A. Smart, E. Garcia, and G. Virk, “Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction,” 2023
work page 2023
-
[14]
Responsible language technologies: Foreseeing and mitigating harms,
S. L. Blodgett, Q. V. Liao, A. Olteanu, R. Mihalcea, M. Muller, M. K. Scheuerman, C. Tan, and Q. Yang, “Responsible language technologies: Foreseeing and mitigating harms,” inExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems , CHI EA ’22, (New York, NY, USA), Association for Computing Machinery, 2022
work page 2022
-
[15]
A framework for understanding sources of harm throughout the machine learning life cycle,
H. Suresh and J. Guttag, “A framework for understanding sources of harm throughout the machine learning life cycle,” in Equity and Access in Algorithms, Mechanisms, and Optimization , EAAMO ’21, ACM, Oct. 2021
work page 2021
-
[16]
A survey of privacy risks and mitigation strategies in the artificial intelligence life cycle,
S. Shahriar, S. Allana, S. M. Hazratifard, and R. Dara, “A survey of privacy risks and mitigation strategies in the artificial intelligence life cycle,”IEEE Access, vol. 11, pp. 61829–61854, 2023
work page 2023
- [17]
-
[18]
Understanding the capabilities, limitations, and societal impact of large language models,
A. Tamkin, M. Brundage, J. Clark, and D. Ganguli, “Understanding the capabilities, limitations, and societal impact of large language models,” 2021
work page 2021
-
[19]
Truth, lies, and automation: How language models could change disinformation,
B. Buchanan, A. Lohn, M. Musser, and K. Sedova, “Truth, lies, and automation: How language models could change disinformation,” May 2021
work page 2021
-
[20]
OpenAI, “Openai usage policies,” 2023.https://openai.com/policies/usage-policies/
work page 2023
-
[21]
Building an early warning system for llm-aided biological threat creation,
OpenAI, “Building an early warning system for llm-aided biological threat creation,” 2024.https://openai.com/ index/building-an-early-warning-system-for-llm-aided-biological-threat-creation/
work page 2024
-
[22]
Deloitte, “Deloitte acquires gryphon scientific business to expand security science and public health capa- bilities,” 2024. https://www2.deloitte.com/us/en/pages/about-deloitte/articles/press-releases/ deloitte-acquires-gryphon-scientific-business-to-expand-security-science-and-public-health-capabilities. html
work page 2024
-
[23]
Sociotechnical safety evaluation of generative ai systems,
L. Weidinger, M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, I. Gabriel, V. Rieser, and W. Isaac, “Sociotechnical safety evaluation of generative ai systems,” 2023
work page 2023
-
[24]
Evaluating and mitigating discrimination in language model decisions,
A. Tamkin, A. Askell, L. Lovitt, E. Durmus, N. Joseph, S. Kravec, K. Nguyen, J. Kaplan, and D. Ganguli, “Evaluating and mitigating discrimination in language model decisions,” 2023
work page 2023
-
[25]
J. A. Goldstein, G. Sastry, M. Musser, R. DiResta, M. Gentzel, and K. Sedova, “Generative language models and automated influence operations: Emerging threats and potential mitigations,” 2023. 29
work page 2023
-
[26]
Exploring relationship development with social chatbots: A mixed-method study of replika,
I. Pentina, T. Hancock, and T. Xie, “Exploring relationship development with social chatbots: A mixed-method study of replika,”Computers in Human Behavior , vol. 140, p. 107600, 2023
work page 2023
-
[27]
Managing extreme ai risks amid rapid progress,
Y. Bengio, G. Hinton, A. Yao, D. Song, P. Abbeel, T. Darrell, Y. N. Harari, Y.-Q. Zhang, L. Xue, S. Shalev-Shwartz, G. Hadfield, J. Clune, T. Maharaj, F. Hutter, A. G. Baydin, S. McIlraith, Q. Gao, A. Acharya, D. Krueger, A. Dragan, P. Torr, S. Russell, D. Kahneman, J. Brauner, and S. Mindermann, “Managing extreme ai risks amid rapid progress,” Science, v...
work page 2024
-
[28]
S. B. Johnson, J. R. Clark, M. C. Luetke, N. M. Butala, A. T. Pearson, J. M. Shapiro, D. M. Aleman, J. M. Lee, M. M. Beil, C. V. Winkle, M. C. Boudreaux, R. C. D’Cunha, H. J. Krouse, and C. Li, “Chatgpt in medical education: a workshop-based large language model-powered intervention for evidence-based clinical decision making in medical students,”Nature M...
work page 2023
-
[29]
Real-world challenges for agi,
K. Kavukcuoglu, “Real-world challenges for agi,” Nov 2021
work page 2021
- [30]
-
[31]
arXiv preprint arXiv:2303.10130 , year=
T. Eloundou, S. Manning, P. Mishkin, and D. Rock, “Gpts are gpts: An early look at the labor market impact potential of large language models,”arXiv preprint arXiv:2303.10130 , 2023
-
[32]
arXiv preprint arXiv:2310.11986 , year=
L. Weidinger, M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, et al., “Sociotechnical safety evaluation of generative ai systems,”arXiv preprint arXiv:2310.11986 , 2023
-
[33]
Wikicrow: Automating synthesis of human scientific knowledge,
S. Cox, M. Hammerling, J. Lála, J. Laurent, S. Rodriques, M. Rubashkin, and A. White, “Wikicrow: Automating synthesis of human scientific knowledge,”Future House, 2023
work page 2023
-
[34]
S. A. Athaluri, S. V. Manthena, V. S. R. K. M. Kesapragada, V. Yarlagadda, T. Dave, and R. T. S. Duddumpudi, “Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through chatgpt references,”Cureus, vol. 15, no. 4, p. e37432, 2023
work page 2023
-
[35]
The dark side of chatgpt: Legal and ethical challenges from stochastic parrots and hallucination,
Z. Li, “The dark side of chatgpt: Legal and ethical challenges from stochastic parrots and hallucination,” 2023
work page 2023
-
[36]
Impact of voice fidelity on decision making: A potential dark pattern?,
M. Dubiel, A. Sergeeva, and L. A. Leiva, “Impact of voice fidelity on decision making: A potential dark pattern?,” 2024
work page 2024
-
[37]
B. Waber, M. Williams, J. S. Carroll, and A. S. Pentland, “A voice is worth a thousand words: The implications of the micro-coding of social signals in speech for trust research,” inHandbook of Research Methods on Trust (G. M. Fergus Lyon and M. N. Saunders, eds.), ch. 23, p. 320, New York: Edward Elgar Publishing, 2011
work page 2011
-
[38]
Friend, mentor, lover: Does chatbot engagement lead to psychological dependence?,
I. Pentina, B. Guo, and W. P. Fan, “Friend, mentor, lover: Does chatbot engagement lead to psychological dependence?,” Journal of Service Management , 2023
work page 2023
-
[39]
Capabilities of GPT-4 on Medical Challenge Problems
H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of gpt-4 on medical challenge problems,” arXiv preprint arXiv:2303.13375 , 2023
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[40]
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? C ase Study in Medicine
H. Nori, Y. T. Lee, S. Zhang, D. Carignan, R. Edgar, N. Fusi, N. King, J. Larson, Y. Li, W. Liu,et al., “Can generalist foundation models outcompete special-purpose tuning? case study in medicine,”arXiv preprint arXiv:2311.16452 , 2023
-
[41]
Large language models encode clinical knowledge,
K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, N. Scharli, A. Chowdhery, P. Mansfield, B. A. y Arcas, D. Webster, G. S. Corrado, Y. Matias, K. Chou, J. Gottweis, N. Tomasev, Y. Liu, A. Rajkomar, J. Barral, C. Semturs, A. Karthikesalingam, and V...
work page 2022
-
[42]
Towards expert-level medical question answering with large language models,
K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, M. Schaek- ermann, A. Wang, M. Amin, S. Lachgar, P. Mansfield, S. Prakash, B. Green, E. Dominowska, B. A. y Arcas, N. Tomasev, Y. Liu, R. Wong, C. Semturs, S. S. Mahdavi, J. Barral, D. Webster, G. S. Corrado, Y. Matias, S. Azizi, A. Karthikesalingam,...
work page 2023
-
[43]
Capabilities of gemini models in medicine,
K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi, J. Z. Chaves, S.-Y. Hu, M. Schaekermann, A. Kamath, Y. Cheng, D. G. T. Barrett, C. Cheung, B. Mustafa, A. Palepu, D. McDuff, L. Hou, T. Golany, L. Liu, J. baptiste Alayrac, N. Houlsby, N. Tomasev, J. Freyberg, C. Lau, J. Kemp, J. Lai, S. Azizi, K. Kanada...
work page 2024
-
[44]
Epic and microsoft bring gpt-4 to ehrs,
Epic Systems Corporation, “Epic and microsoft bring gpt-4 to ehrs,”Epic, 2023
work page 2023
-
[45]
Adapted large language models can outperform medical experts in clinical text summarization,
D. Van Veen, C. Van Uden, L. Blankemeier, J.-B. Delbrouck, A. Aali, C. Bluethgen, A. Pareek, M. Polacin, E. P. Reis, A. Seehofnerová, et al., “Adapted large language models can outperform medical experts in clinical text summarization,” Nature medicine, vol. 30, no. 4, pp. 1134–1142, 2024
work page 2024
-
[46]
Epic and microsoft bring gpt-4 to ehrs,
Epic, “Epic and microsoft bring gpt-4 to ehrs,” 2023. 30
work page 2023
-
[47]
Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages,
P. Garcia, S. P. Ma, S. Shah, M. Smith, Y. Jeong, A. Devon-Sand, M. Tai-Seale, K. Takazawa, D. Clutter, K. Vogt, C. Lugtu, M. Rojo, S. Lin, T. Shanafelt, M. A. Pfeffer, and C. Sharp, “Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages,”JAMA Network Open , vol. 7, pp. e243201–e243201, 03 2024
work page 2024
-
[48]
Paradigm: Improving patient access to clinical trials
OpenAI, “Paradigm: Improving patient access to clinical trials.”https://openai.com/index/paradigm/, 2024. Accessed: 2024-08-07
work page 2024
-
[49]
How ai is being used to accelerate clinical trials,
M. Hutson, “How ai is being used to accelerate clinical trials,”Nature, vol. 627, pp. S2–S5, 2024
work page 2024
-
[50]
Using gpt-4o reasoning to transform cancer care
OpenAI, “Using gpt-4o reasoning to transform cancer care.”https://openai.com/index/color-health/, 2024. Accessed: 2024-08-07
work page 2024
-
[51]
Systematic analysis of chatgpt, google search and llama 2 for clinical decision support tasks,
J. Varghese and J.-L. Chapiro, “Systematic analysis of chatgpt, google search and llama 2 for clinical decision support tasks,”Nature Communications, vol. 15, no. 1, p. 46411, 2024. Accessed: 2024-08-07
work page 2024
-
[52]
E. Schmidt, “Ai will transform science.” https://www.technologyreview.com/2023/07/05/1075865/ eric-schmidt-ai-will-transform-science/ , 2023. Accessed: 2024-08-07
work page 2023
-
[53]
Science, invention and economic growth,
N. Rosenberg, “Science, invention and economic growth,”The Economic Journal , vol. 84, no. 333, pp. 90–108, 1974
work page 1974
-
[54]
The dual-use dilemma for the life sciences: Perspectives, conundrums, and global solutions,
R. M. Atlas and M. Dando, “The dual-use dilemma for the life sciences: Perspectives, conundrums, and global solutions,” Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science , vol. 4, no. 3, pp. 276–286, 2006. PMID: 16999588
work page 2006
-
[55]
X. Gu and M. Krenn, “Generation and human-expert evaluation of interesting research ideas using knowledge graphs and large language models,” 2024
work page 2024
-
[56]
A. Ghafarollahi and M. J. Buehler, “Atomagents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence,” 2024
work page 2024
-
[57]
Lab-bench: Measuring capabilities of language models for biology research,
J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, and S. G. Rodriques, “Lab-bench: Measuring capabilities of language models for biology research,” 2024
work page 2024
-
[58]
Sciassess: Benchmarking llm proficiency in scientific literature analysis,
H. Cai, X. Cai, J. Chang, S. Li, L. Yao, C. Wang, Z. Gao, H. Wang, Y. Li, M. Lin, S. Yang, J. Wang, M. Xu, J. Huang, F. Xi, J. Zhuang, Y. Yin, Y. Li, C. Chen, Z. Cheng, Z. Zhao, L. Zhang, and G. Ke, “Sciassess: Benchmarking llm proficiency in scientific literature analysis,” 2024
work page 2024
-
[59]
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the AI2 reasoning challenge,”CoRR, vol. abs/1803.05457, 2018
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[60]
TruthfulQA: Measuring How Models Mimic Human Falsehoods
S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” CoRR, vol. abs/2109.07958, 2021. A Violative & Disallowed Content - Full Evaluations We used TTS to convert existing text safety evals to audio. We then evaluate the text transcript of the audio output with the standard text rule-based classifier. Our two main metr...
work page internal anchor Pith review Pith/arXiv arXiv 2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.