fix(pdf): Fix garbled text in PDF loaders

[engkimo]

Hi team,

This PR implements the first part of the fix for the garbled text issue (#29555), starting with PDF loaders. This is part of a larger effort to address encoding issues across multiple document loaders.

Changes

• Add PDFEncoding enum for standardized encoding options
• Implement PDFEncodingConfig for centralized encoding management
• Enhance BasePDFLoader with encoding configuration support
• Update PDFMinerParser to handle encoding properly
• Add fallback mechanism for encoding errors

Next Steps

This PR focuses on PDF loaders as the first implementation. Similar fixes will be needed for:

• Docx2txtLoader
• PyPDFLoader
• TextLoader
• UnstructuredExcelLoader
• UnstructuredWordDocumentLoader

Notes

• No breaking changes to existing API
• Default encoding remains UTF-8
• Users can now specify custom encoding configurations when needed
• Fixes regression introduced in o3 release for PDF loaders

Environment

• Tested on Python 3.13.1
• Compatible with:
  • langchain 0.3.17
  • langchain_community 0.3.16
  • langchain_core 0.3.33

Related issue: #29555

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Feb 3, 2025 3:37pm