Spark NLP

Health v2 · maintenance only

Adoption not part of the score

4159 stars · 743 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Spark NLP is an open-source natural language processing library built natively on Apache Spark, providing scalable NLP annotations and transformer models (BERT, T5, Whisper, Llama, etc.) for Python, Scala, Java, and R. It ships with 100,000+ pretrained pipelines and models in 200+ languages for tasks like NER, sentiment analysis, translation, summarization, and question answering.

Use cases

• run named entity recognition on large text corpora at scale
• perform sentiment analysis in a distributed Spark pipeline
• translate text across 180+ languages
• summarize documents with transformer models
• deploy BERT or Llama models in production on the JVM
• transcribe audio with Whisper in a Spark job
• classify documents and detect entities in enterprise data pipelines

When to choose

• you need NLP at scale on big data using Apache Spark clusters
• you want production-ready pretrained models in many languages
• your team works in Python, Scala, or Java on the JVM
• you need GPU-accelerated transformer inference in distributed pipelines

When to avoid

• you need lightweight single-machine NLP without a Spark dependency
• you want to fine-tune cutting-edge LLMs with the latest research tooling
• your project is small enough that Spark's overhead outweighs its benefits

Facets

library · maturity stable

nlp machine-learning llm-inference speech-recognition image-processing rag machine-learning large-language-models data-science big-data python jvm cloud apache-spark transformers pretrained-models distributed-computing named-entity-recognition sentiment-analysis onnx tensorflow natural-language-processing gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="JohnSnowLabs/spark-nlp")