CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Pith reviewed 2026-05-12 16:00 UTC · model grok-4.3
The pith
Releasing the CodeSearchNet Corpus of 6 million functions and a challenge with 99 annotated queries enables evaluation of semantic code search across six languages.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By releasing the CodeSearchNet Corpus containing approximately 6 million functions across six programming languages together with automatically generated natural-language descriptions for two million of them, and by creating the CodeSearchNet Challenge of 99 natural-language queries annotated with roughly four thousand expert relevance judgments, the work supplies a standardized corpus and evaluation set for semantic code search.
What carries the argument
The CodeSearchNet Corpus of functions paired with scraped documentation text, plus the expert-annotated query set used to score retrieval relevance.
Load-bearing premise
Mechanically scraped and preprocessed function documentation yields sufficiently accurate and representative natural-language queries and the expert annotations are consistent and unbiased measures of relevance.
What would settle it
An independent check finding low agreement among experts on the same query-code pairs or showing that the scraped documentation text rarely matches how developers actually phrase searches would undermine the benchmark.
read the original abstract
Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to release the CodeSearchNet Corpus containing about 6 million functions from six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby), with automatically generated query-like natural language descriptions for 2 million functions obtained by mechanically scraping and preprocessing associated documentation. It also introduces the CodeSearchNet Challenge consisting of 99 natural language queries with about 4k expert relevance annotations of likely results from the corpus, describes the methodology for corpus construction and labeling, and provides a number of simple baseline solutions for semantic code search, with the aim of enabling evaluation and tracking of progress via a future competition and leaderboard.
Significance. If the generated queries and expert annotations prove reliable and representative, the release of this large-scale multi-language corpus and annotated challenge would be a significant contribution to semantic code search research. It provides a standardized benchmark at a scale (6M functions, 99 queries, 4k annotations) that could facilitate model development and comparison in bridging natural language and code, similar to other information retrieval benchmarks, and the inclusion of baselines supports immediate usability.
major comments (2)
- [Methodology for corpus and query generation] The methodology section describing corpus construction and query generation states that the 2 million query-like descriptions are obtained by mechanically scraping and preprocessing function documentation, but provides no quantitative validation (e.g., accuracy against human queries, diversity metrics, or fidelity checks) to support that these yield sufficiently accurate and representative natural language queries for the challenge.
- [Challenge construction and expert annotation process] The section on the CodeSearchNet Challenge and expert labels describes the 99 queries and ~4k annotations but reports no inter-annotator agreement statistics, annotation consistency measures, or bias validation, which is load-bearing for the central claim that the challenge enables reliable evaluation of semantic code search progress.
minor comments (2)
- [Abstract and conclusion] The abstract and conclusion mention plans to host a competition and leaderboard but do not specify the platform, timeline, or evaluation protocol details.
- [Baseline solutions] The baselines are introduced as 'simple' solutions; adding implementation details or pseudocode would improve reproducibility without altering the core contribution.
Simulated Author's Rebuttal
We thank the referee for the constructive review. We appreciate the opportunity to clarify the distinction between the corpus construction and the challenge, and to address the points on validation and annotation reliability.
read point-by-point responses
-
Referee: [Methodology for corpus and query generation] The methodology section describing corpus construction and query generation states that the 2 million query-like descriptions are obtained by mechanically scraping and preprocessing function documentation, but provides no quantitative validation (e.g., accuracy against human queries, diversity metrics, or fidelity checks) to support that these yield sufficiently accurate and representative natural language queries for the challenge.
Authors: We thank the referee for raising this. We wish to clarify an important distinction: the 2 million automatically generated descriptions are part of the released CodeSearchNet Corpus and serve as weak supervision for training models; they are obtained by scraping and preprocessing existing function documentation from open-source repositories. These descriptions are not the queries used in the CodeSearchNet Challenge. The challenge instead uses a separate set of 99 expert-written natural language queries, each paired with expert relevance annotations over candidate functions from the corpus. Because the corpus descriptions are mechanically derived from documentation that already exists in the source code, their representativeness is bounded by the quality of that documentation, which we describe in the methodology section. We did not include additional quantitative validation (such as diversity metrics or fidelity checks against human queries) because the primary contribution is the release of the large-scale resource itself rather than a claim that the derived descriptions perfectly match human queries. We can, however, add basic descriptive statistics on the generated descriptions (e.g., length distributions and language-specific characteristics) in a revision to improve transparency. revision: partial
-
Referee: [Challenge construction and expert annotation process] The section on the CodeSearchNet Challenge and expert labels describes the 99 queries and ~4k annotations but reports no inter-annotator agreement statistics, annotation consistency measures, or bias validation, which is load-bearing for the central claim that the challenge enables reliable evaluation of semantic code search progress.
Authors: We acknowledge that reporting inter-annotator agreement would strengthen the perceived reliability of the annotations. The 99 queries were authored by the paper authors and a small group of domain experts, and the approximately 4k relevance judgments were performed by the same expert annotators following a written annotation protocol that specified relevance criteria, handling of edge cases, and tie-breaking rules. Due to the expert-only nature of the task and resource limitations, we collected only a single annotation per query–function pair and therefore do not have the data required to compute standard IAA metrics such as Cohen’s kappa or Fleiss’ kappa. We attempted to mitigate bias and inconsistency through careful query curation, pilot annotation rounds, and discussion of difficult cases among annotators. We agree this is a limitation of the current release. In the revised manuscript we will expand the description of the annotation protocol, explicitly note the absence of multiple annotations, and discuss the implications for benchmark reliability. revision: yes
Circularity Check
No circularity: dataset release paper with no derivations or self-referential reductions
full rationale
The paper's core contribution is the release of the CodeSearchNet Corpus (6M functions across 6 languages) and Challenge (99 NL queries with ~4k expert annotations), obtained by scraping documentation for 2M functions and expert labeling. No equations, fitted parameters, predictions, or derivation chains appear in the abstract or described methodology. Claims about enabling evaluation of semantic code search progress rest on the data collection process itself rather than any mathematical reduction to prior inputs. No self-citations or ansatzes are invoked as load-bearing steps. This matches the expected non-circular outcome for a pure data/benchmark paper.
Axiom & Free-Parameter Ledger
Forward citations
Cited by 60 Pith papers
-
Architecture Determines Observability of Transformers
Certain transformer architectures lose internal linear signals for decision quality during training, making observability an architecture-dependent property rather than a universal one.
-
Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning
Introduces SolidityBench benchmark and SolidityScore metric for repository-level Solidity code generation, finding supervised fine-tuning outperforms prompting, CoT, ICL, and RAG methods on evaluated LLMs.
-
AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model
AMALIA-VL introduces the first open-source instruction-tuned LVLM natively optimized for European Portuguese via vision-language alignment, instruction tuning, preference optimization, and a pt-PT-centric data mix.
-
Signature filtering: a lightweight enhancement for statistical watermark detection in large language models
Signature filtering learns unreliable tokens with MILP and removes them at detection time, raising true positive rates from 8-31% to 78-99% across Kgw, Sweet, Unigram, and Exp watermarks on multiple corpora and LLMs w...
-
Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?
On heterogeneous document collections, only query expansion and a newly introduced per-source calibrated corrector (SSCC) deliver reliable gains beyond a strong cross-encoder reranker; other common retrieval enhanceme...
-
HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection
HybridCodeAuthorship is a new benchmark dataset of interleaved human-AI Python code that shows existing detection algorithms reach at most 0.48 F1 at chunk level and 0.56 F1 at line level.
-
CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding
CORE-Bench is a benchmark for code retrieval in agentic coding settings, built from curated tasks and SWE-bench instances, showing performance drops and gains from fine-tuning.
-
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
SWE-Explore is a new benchmark evaluating repository exploration by coding agents on 848 issues across 203 repositories, using line-level ground truth from successful agent trajectories and showing agentic methods out...
-
VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
VISTA is a new benchmark for end-to-end visual spec-to-web-app generation by LLM agents, featuring five prompt conditions, manual UI annotations, multi-metric evaluation, and results on four agent systems showing part...
-
Test-Time Speculation
Test-Time Speculation adapts draft models online via target-model verifications to sustain high acceptance lengths during long LLM generations.
-
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Sinks are equivalent to hard attention switches that zero out outputs and are cheaper than diagonal patterns when self-communication is allowed, closing the gap between oversmoothing prevention needs and what sinks provide.
-
Evaluating Non-English Developer Support in Machine Learning for Software Engineering
Code LLMs generate substantially worse comments outside English, and no tested automatic metric or LLM judge reliably matches human assessment of those outputs.
-
An Empirical Study of Proactive Coding Assistants in Real-World Software Development
Real developer IDE traces differ substantially from LLM simulations in behavior and structure; current proactive assistants are unreliable on real traces, and simulated data cannot substitute for real data in training.
-
POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference
POSTCONDBENCH is a new multilingual benchmark that evaluates LLM postcondition generation on real code using defect discrimination to assess completeness beyond surface matching.
-
PuzzleMark: Implicit Jigsaw Learning for Robust Code Dataset Watermarking in Neural Code Completion Models
PuzzleMark provides a robust and imperceptible watermarking method for code datasets using adaptive variable name concatenation and statistical verification, achieving perfect detection rates with minimal performance impact.
-
RepoDoc: A Knowledge Graph-Based Framework to Automatic Documentation Generation and Incremental Updates
RepoDoc uses a repository knowledge graph with module clustering and semantic impact propagation to generate more complete documentation 3x faster with 85% fewer tokens and handle incremental updates 73% faster than p...
-
CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
CodeMMR creates a unified embedding space for text, code, and images, outperforming baselines by 10 nDCG@10 points and boosting RAG code generation quality.
-
Can LLMs Deobfuscate Binary Code? A Systematic Analysis of Large Language Models into Pseudocode Deobfuscation
LLM deobfuscation of binaries to pseudocode depends more on reasoning ability and task-specific fine-tuning than on model size, with reasoning models showing robustness across ISAs and obfuscation levels on the new Bi...
-
When RL Meets Adaptive Speculative Training: A Unified Training-Serving System
Aurora unifies speculative decoder training and serving via asynchronous RL on inference traces, delivering 1.5x day-0 speedup on frontier models and 1.25x adaptation gains on distribution shifts.
-
OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research
OpenClassGen supplies 324,843 real-world Python classes with self-contained skeletons and static metrics to support LLM class generation research and evaluation.
-
InCoder: A Generative Model for Code Infilling and Synthesis
InCoder is the first generative model to directly perform zero-shot code infilling via bidirectional context from a masked-then-appended training scheme, matching left-to-right models on synthesis while improving on t...
-
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
CodeBLEU improves correlation with human programmer scores on code synthesis tasks by adding syntactic AST matching and semantic data-flow matching to the standard BLEU n-gram approach.
-
GraphCodeBERT: Pre-training Code Representations with Data Flow
GraphCodeBERT uses data flow graphs in pre-training to capture semantic code structure and reaches state-of-the-art results on code search, clone detection, translation, and refinement.
-
The Decomposition Is the Fingerprint: Per-Component Identity for Agent Skills
A per-component SimHash fingerprint supplies structural identity for AI agent skills, recovering family membership under paraphrase and refactoring with AUC 0.974 while localizing changes.
-
AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model
AMALIA-VL is the first open-source LVLM natively optimized for European Portuguese via three-stage training on a pt-PT-centric data mix combining curated, translated, and novel datasets.
-
AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model
Introduces AMALIA-VL, the first open-source instruction-tuned LVLM for European Portuguese, using a high-resolution vision encoder, pt-PT language model, learned connector, and three-stage training on a custom data mix.
-
Securing Code Understanding: Detecting Natural Backdoor Vulnerability in Code Language Models
Natural backdoors are prevalent in CodeLMs; the authors propose ScanNBT to detect them after analyzing differences from injected backdoors, transferability, and causes.
-
UniRTL: Unifying Code and Graph for Robust RTL Representation Learning
UniRTL unifies RTL code and CDFG through mutual masked modeling and hierarchical training with a graph-aware tokenizer, outperforming prior single-modality methods on performance prediction and code retrieval.
-
Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
Code-QA-Bench uses an answer-first pipeline and three-condition experiments to generate 628 tasks across 10 Python repositories and quantify that code access drives most performance gains while documentation adds only...
-
Strong Teacher Not Needed? On Distillation in LLM Pretraining
Even small or undertrained teachers improve larger LLM students via distillation with tuned loss mixing, while stronger teachers can saturate or reverse gains and distillation aids generalization more than in-domain fit.
-
Improving BM25 Code Retrieval Under Fixed Generic Tokenization: Adaptive q-Log Odds as a Drop-In BM25 Fix
A q-log odds variant of BM25 raises NDCG@10 by 89% relative on CodeSearchNet Go under fixed generic tokenization while recovering standard BM25 at q=1.
-
XSearch: Explainable Code Search via Concept-to-Code Alignment
XSearch achieves explainable code search by breaking queries into functional concepts and matching them directly to code statements, delivering large gains on out-of-distribution benchmarks.
-
XSearch: Explainable Code Search via Concept-to-Code Alignment
XSearch achieves 15x gains on out-of-distribution code search benchmarks by replacing global embedding similarity with explicit concept-to-statement alignment.
-
Test-Time Speculation
TTS adapts speculator models online via target model verifications to improve acceptance lengths by up to 72% over prior methods, with gains increasing for longer generations.
-
Do not copy and paste! Rewriting strategies for code retrieval
Full natural-language rewriting of code and queries boosts retrieval on code benchmarks while corpus-only rewriting often hurts, with token entropy difference serving as a cheap predictor of gains.
-
A Survey of Reasoning-Intensive Retrieval: Progress and Challenges
A survey that categorizes RIR benchmarks by domain and modality, proposes a taxonomy for integrating reasoning into retrieval pipelines, and outlines key challenges.
-
VulStyle: A Multi-Modal Pre-Training for Code Stylometry-Augmented Vulnerability Detection
VulStyle pre-trains on 4.9M functions using code, non-terminal ASTs, and stylometry features, then fine-tunes to achieve SOTA F1 gains of 4-48% on BigVul and VulDeePecker.
-
Architecture Determines Observability of Transformers
Architecture and training determine whether transformers retain a readable internal signal that lets activation monitors catch errors missed by output confidence.
-
Less Is More: Measuring How LLM Involvement affects Chatbot Accuracy in Static Analysis
A structured JSON intermediate representation for LLM-generated static analysis queries outperforms both direct generation and agentic tool use, with gains of 15-25 percentage points on large models.
-
DuCodeMark: Dual-Purpose Code Dataset Watermarking via Style-Aware Watermark-Poison Design
DuCodeMark watermarks code datasets using AST style transformations and repressible poisons for both source-code and decompilation tasks, verified by t-test, with high stealth and a 28.6% performance drop if removed.
-
On the Role of Fault Localization Context for LLM-Based Program Repair
More fault localization context does not consistently improve LLM-based program repair; file-level context gives 15-17x gains, optimal around 6-10 files, while line-level context often degrades performance from noise.
-
A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?
Student models distilled from code language models often fail to deeply mimic teachers, showing up to 62% behavioral discrepancies and 285% worse drops under attacks that accuracy metrics miss.
-
Understanding Robustness of Model Editing in Code LLMs
A controlled benchmark on 2040 problems reveals poor generalization and high interference in model editing for API updates in code LLMs, with many successes being workarounds rather than true migrations.
-
PseudoBridge: Pseudo Code as the Bridge for Better Semantic and Logic Alignment in Code Retrieval
PseudoBridge uses LLM-synthesized pseudo-code to bridge NL semantics and PL logic plus logic-invariant style augmentation to boost robustness and generalization in code retrieval.
-
LinkAnchor: An Autonomous LLM-Based Agent for Issue-to-Commit Link Recovery
LinkAnchor is an LLM-based autonomous agent with lazy-access architecture for recovering issue-to-commit links in software repositories.
-
Fine-Tuning Code Language Models to Detect Cross-Language Bugs
Fine-tuning 13 CodeLMs on a constructed CLB dataset with nine interaction types improves detection, with UniXcoder-base reaching F1 0.7407 and small models outperforming large ones.
-
XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants
XOXO is a cross-origin context poisoning attack on AI coding assistants that uses a Cayley Graph search algorithm (GCGS) to find stealthy perturbations, achieving 75.72% average success rate across five tasks and elev...
-
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
LiveCodeBench collects 400 recent contest problems to create a contamination-free benchmark evaluating LLMs on code generation and related capabilities like self-repair and execution.
-
Nomic Embed: Training a Reproducible Long Context Text Embedder
Nomic AI produced and open-sourced a reproducible 8192-context English text embedder that exceeds OpenAI Ada-002 and text-embedding-3-small performance on MTEB short-context and LoCo long-context benchmarks.
-
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
CRUXEval benchmark shows current code models including GPT-4 achieve at most 81% on input and output prediction for short Python functions, exposing gaps not captured by HumanEval.
-
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
CodeT5+ is a flexible encoder-decoder LLM family for code pretrained with diverse objectives on multilingual corpora and initialized from existing LLMs, achieving state-of-the-art results on code generation, completio...
-
Text and Code Embeddings by Contrastive Pre-Training
Contrastive pre-training on unsupervised data at scale creates text and code embeddings that set new state-of-the-art results on classification and semantic search benchmarks.
-
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
CodeT5 adds identifier-aware pre-training and bimodal dual generation to a T5-style encoder-decoder, yielding better results on defect detection, clone detection, and code-to-text, text-to-code, and code-to-code tasks...
-
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
CodeXGLUE supplies a standardized collection of 10 code-related tasks, 14 datasets, an evaluation platform, and BERT-, GPT-, and encoder-decoder-style baselines.
-
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
CodeBERT pre-trains a bimodal model on code and text pairs plus unimodal data to achieve state-of-the-art results on natural language code search and code documentation generation.
-
Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebases
Agent4cs deploys summarization, keyword-extraction, and quality-assurance agents in a bottom-up pipeline that raises semantic consistency by 8% and normalized keyword coverage by up to 38% over structured prompting ba...
-
RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
RESOURCE2SKILL converts multimodal human resources into a hierarchical Skill Wiki of executable agent skills, reporting +11.9 percentage point average gains over no-skill baselines across seven authoring domains.
-
UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer Learning
UNICS pre-trains on a pseudocode dataset for cross-lingual logic then applies multi-task transfer learning with hard-positive mining and dynamic hard-negative sampling to reach claimed SOTA on multilingual code-search...
-
JupOtter: Cell-Level Bug Detection in Jupyter Notebooks
JupOtter introduces notebook-specific tokenization, cell-level bug prediction, and OtterDataset to achieve higher F1 scores than static analyzers and LLMs on two of three evaluation datasets.
-
ConcernBERT: Learning Responsibilities Using Class Membership
ConcernBERT is a BERT embedding model trained with triplet loss on class membership to encode concern-level semantics in Java entities, evaluated by recovering original classes from merged unlabeled groups on a new da...
Reference graph
Works this paper leans on
- [1]
-
[2]
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018), 81
work page 2018
-
[3]
Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. In Proceedings of the International Conference on Machine Learning (ICML)
work page 2016
-
[4]
Uri Alon, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018)
work page Pith review arXiv 2018
-
[5]
Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of Python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275 (2017)
work page Pith review arXiv 2017
-
[6]
Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra
-
[7]
arXiv preprint arXiv:1905.03813 (2019)
When Deep Learning Met Code Search. arXiv preprint arXiv:1905.03813 (2019)
-
[8]
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Ben- gio. 2014. On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. Syntax, Semantics and Structure in Statistical Translation (2014)
work page 2014
-
[9]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv CodeSearchNet Encoder CodeSearchNet Challenge– NDCG Within CodeSearchNet Challenge– NDCG All Text Code Go Java JS PHP Python Ruby Avg Go Java JS PHP Python Ruby Avg ElasticSearch 0.307 0.257 0.318 0...
work page internal anchor Pith review Pith/arXiv arXiv 2018
- [10]
-
[11]
Philip Gage. 1994. A new algorithm for data compression. The C Users Journal 12, 2 (1994), 23–38
work page 1994
-
[12]
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE) . IEEE, 933–944
work page 2018
-
[13]
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API Learning. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE)
work page 2016
-
[14]
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems . 10073–10083
work page 2018
-
[15]
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Map- ping language to code in programmatic context. arXiv preprint arXiv:1808.09588 (2018)
work page Pith review arXiv 2018
-
[16]
Yoon Kim. 2014. Convolutional neural networks for sentence classification.arXiv preprint arXiv:1408.5882 (2014)
work page Pith review arXiv 2014
-
[17]
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. In International Conference on Language Resources and Evaluation
work page 2018
-
[18]
Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent Predictor Networks for Code Generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)
work page 2016
-
[19]
Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: a map of code duplicates on GitHub. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 84
work page 2017
-
[20]
Manning, Prabhakar Raghavan, and Hinrich Schütze
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Intro- duction to Information Retrieval . Cambridge University Press
work page 2008
-
[21]
Bhaskar Mitra, Nick Craswell, et al. 2018. An introduction to neural information retrieval. Foundations and Trends® in Information Retrieval 13, 1 (2018), 1–126
work page 2018
-
[22]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. InProceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)
work page 2016
-
[23]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems . 5998–6008
work page 2017
-
[24]
Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. 2019. CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning. (2019)
work page 2019
-
[25]
Ziyu Yao, Daniel S Weld, Wei-Peng Chen, and Huan Sun. 2018. StaQC: A Sys- tematically Mined Question-Code Dataset from Stack Overflow. InProceedings of the 2018 World Wide Web Conference on World Wide Web . International World Wide Web Conferences Steering Committee, 1693–1703
work page 2018
-
[26]
Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General- Purpose Code Generation. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)
work page 2017
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.