theseer/tokenizer

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

5195 stars · 24 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A small PHP library that tokenizes PHP source code and serializes the resulting tokens into XML (and potentially other formats). It is commonly used as a dependency in PHP tooling such as PHPUnit's code coverage reporting.

Use cases

• convert PHP source code tokens to XML
• analyze PHP source code structure programmatically
• generate XML representations of tokenized PHP files
• build code coverage or static analysis tools for PHP
• process PHP tokens for custom tooling

When to choose

• you need to serialize tokenized PHP source into XML
• you are building PHP code analysis or coverage tooling
• you need a lightweight, Composer-installable tokenizer utility

When to avoid

• you need to tokenize languages other than PHP
• you need a full static analysis or AST tool rather than token-level output
• you need JSON or other output formats without writing your own serializer

Facets

library · maturity stable

parser serialization developer-tools developer-tools parsers php cross-platform tokenizer xml-serialization php-source-code code-analysis php

Member repositories

For agents

markdown · JSON · MCP: product_card(name="theseer/tokenizer")