Handling the unrecognizable characters in Stanford Parser

Handling the unrecognizable characters in Stanford Parser/Tokenizer for warnings like :
WARNING: Unrecognizable .... U+FFD ... for some character.
You may find solution here :
http://nlp.stanford.edu/software/tokenizer.shtml
i.e. untokenizable: What to do with untokenizable characters (ones not known to the tokenizer). Six options combining whether to log a warning for none, the first, or all, and whether to delete them or to include them as single character tokens in the output: noneDelete, firstDelete, allDelete, noneKeep, firstKeep, allKeep. The default is "firstDelete".

Comments

Publishing business basics

Basic Steps: 1. Decide name for the company 2. Register the company with ministry - you will need an attorney (Lawyer for that) 3. Register with Registrar of News Papers in India if it's a magazine/News paper. 4. Study the relevant acts in general or get them known from the lawyer 5. Start publishing Following are details regarding the same (not that well written) : ----- Some starts and books; * Start Your Own Self-Publishing Business (Entrepreneur Magazine's Start Up) by Entrepreneur Press * How To Start And Run A Small Book Publishing Company: A Small Business Guide To Self-Publishing And Independent Publishing by Peter I. Hupalo * Art & Science Of Book Publishing by Herbert S., Jr. Bailey * This Business of Books: A Complete Overview of the Industry from Concept Through Sales by Claudia Suzanne Raja Rammohun Roy National Agency for ISBN West Block-I, Wing-6, 2nd Floor, Sector -I, R.K. Puram, New Delhi-110066 Some new things and t...

ती अशीच त्याला भेटली, अगदी अचानक आलेल्या पावसासारखी

खूप पावसाळे गेले. पण अगदी अलीकडे पर्यंत त्याला पाऊस कधीच आवडलेला नव्हता. पाऊस म्हंटल की त्याला भीती वाटायची, नसती कट कट वाटायची. भिजणे तर त्याला कधीच नको वाटायचे, उगाच सर्दी ला आमंत्रण. पण मागच्या पवसाळ्यात ढग असे आले आणि पाऊस इतका पडला की तो पावसाच्या प्रेमातच पडला. पाऊस आधीही पडायचा, कदाचित असाच, इतकाच किंवा जास्त ही. पण या वेळेस मात्र पाऊस एकटा आला नाही, त्याच्या आयुष्यात सोबत घेऊन आला तिच्या केसांचा ओला सुगंध. ती अशीच त्याला भेटली, अगदी अचानक आलेल्या मुंबईतल्या पावसासारखी, चिंब भिजलेली. त्याला ती आवडली आणि म्हणून पाऊस ही. ती त्याला पुन्हा भेटली, न ठरवून, अशीच पुन्हा एकदा अचानक पावसाराखीच. ते बोलले, थांबले, विसावले. ती समोर आली तेंव्हा पाऊस नव्हताच, ती अजून थोडी जवळ आली तेंव्हा ही पाऊस नव्हता. तो तिच्याकडे सारखा पहातच होता, पण नजर चोरून. आयुष्यात पहिल्यांदा त्याने पावसाकडे येण्याची विनवणी केली. आशेच्या नजरेने त्याने ढगांकडे पाहिले. ढग होते, पण अजून पाऊस मात्र नव्हता. तरी ही आजूबाजूंच्या झाडात, डोंगरात, फुलात आणि पाखरात ओलावा होताच. त्याच्या विनवनीने अखेर तो आला. रिम-झिम, रिम-झिम...

NVDIA AI India

NVDIA AI some Insights after digesting the event with some Diwali Delicacies! I had good time at the NVIDIA AI Summit held at the Jio World Center in Mumbai. It felt like every company working on artificial intelligence in India was present, either as an exhibitor or an attendee. Some sessions were so packed that people were standing, and even then, many more were trying to get in. Much of the discussion revolved around Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG). These are undoubtedly central topics in AI today, drawing significant attention. I’m a huge admirer of RAG, but I still wouldn’t go as far as to say that “LLM (+RAG) is AI.” Although no one at the conference explicitly said this, it felt implied over the three days of sessions. I may be wrong, but I sensed a push for these technologies. Not just from NVIDIA, but from any hardware supplier, there’s an incentive to promote anything that drives demand for their solutions. NVIDIA’s GPUs are a backbo...

The Personal Blog

Search This Blog