pub fn tokenize(src: &str) -> Result<Vec<Token>, LexError>Expand description
Tokenize a complete HLSL source string. Returns the full token stream
terminated by TokenKind::Eof, or the first error encountered.
The lexer never panics on user input — every fall-through case becomes
either a real token or a LexError. Comments and #-prefixed
preprocessor lines are skipped silently.