Added more samples and additional GPT-generated text to the prelude file

Signed-off-by: Jacob Torrey <jacob@thinkst.com>

ai-generated.txt

@@ -95,6 +95,36 @@ Common phrases hold a special place in our language and culture. They capture th
 
 35 year old male presents with complaint of head and neck pain. The pain is described as a constant dull ache, with occasional sharp shooting pains. The patient reports the pain is worse when he moves his head and neck, and it interferes with his ability to work and perform daily activities. The patient denies any trauma or injury to the head or neck. The pain started about a week ago and has not improved despite over the counter pain medication. The patient has a history of migraines, but this pain feels different. On physical examination, the patient has tenderness to palpation over the neck muscles and limited range of motion in the neck. The rest of the physical examination is within normal limits. The patient is started on a muscle relaxant and referred for physical therapy. Follow up in one week for reevaluation of symptoms.
 
+Summary of Medical Guidance for COVID-19
+
+1. Vaccination: COVID-19 vaccination is strongly recommended for eligible individuals. Vaccines have proven to be safe and effective in reducing the risk of severe illness, hospitalization, and death. Follow the guidelines of your local health authorities regarding eligibility, scheduling, and specific vaccine recommendations.
+
+2. Prevention Measures: To reduce the spread of COVID-19, it is essential to follow preventive measures, including:
+
+   a. Wearing Masks: Wear a mask in public settings, especially indoors or when social distancing is challenging.
+   
+   b. Hand Hygiene: Wash your hands frequently with soap and water for at least 20 seconds. If soap and water are not available, use hand sanitizer with at least 60% alcohol.
+   
+   c. Social Distancing: Maintain a distance of at least 1 meter (3 feet) from others, especially if they are coughing, sneezing, or not wearing masks.
+   
+   d. Respiratory Etiquette: Cover your mouth and nose with a tissue or your elbow when coughing or sneezing. Dispose of used tissues properly and wash your hands immediately.
+   
+   e. Ventilation: Ensure proper ventilation in enclosed spaces by opening windows or using air conditioning systems.
+   
+   f. Avoid Crowded Places: Minimize attendance at large gatherings, crowded events, and poorly ventilated areas.
+
+3. Testing and Monitoring: If you experience symptoms suggestive of COVID-19, such as fever, cough, sore throat, loss of taste or smell, or difficulty breathing, get tested as per local guidelines. Follow the advice of healthcare professionals regarding self-isolation and monitoring of symptoms.
+
+4. Quarantine and Isolation: Adhere to quarantine and isolation guidelines if you are diagnosed with COVID-19, have been in close contact with someone infected, or have recently traveled from a high-risk area. Follow the instructions of healthcare providers and health authorities regarding the duration and precautions during quarantine or isolation.
+
+5. Treatment: Currently, several treatments are available for COVID-19. If diagnosed with COVID-19, consult healthcare professionals for appropriate management options, which may include antiviral medications, immune-based therapies, and supportive care.
+
+6. Post-COVID Recovery: Some individuals may experience long-term symptoms even after recovering from acute COVID-19 infection (known as long COVID or post-acute sequelae of SARS-CoV-2 infection). If you continue to experience persistent symptoms, such as fatigue, shortness of breath, or cognitive difficulties, seek medical attention for evaluation and support.
+
+7. Stay Informed: Stay updated with reliable sources of information, such as local health authorities, the World Health Organization (WHO), and the Centers for Disease Control and Prevention (CDC). Follow their guidelines, recommendations, and any updates regarding COVID-19 prevention, testing, and treatment.
+
+It is important to note that the guidance and recommendations for COVID-19 may vary based on the evolving understanding of the virus and local epidemiological conditions. Therefore, always refer to the most recent guidelines provided by your local health authorities for accurate and up-to-date information.
+
 Unprecedented Weather Patterns Sweep Across United States, Bringing Challenges and Surprises
 
 In a startling turn of events, the United States finds itself in the grip of a series of unprecedented weather patterns, causing both awe and concern among its residents. From coast to coast, this meteorological rollercoaster has brought about unexpected conditions and posed unique challenges for communities across the country.
@@ -154,4 +184,257 @@ Space exploration plays a vital role in the search for extraterrestrial life. Mi
 
 Conclusion:
 
-Space exploration continues to inspire and ignite our collective imagination. It is a testament to humanity's insatiable curiosity and our desire to unravel the mysteries of the cosmos. Through scientific discoveries, technological innovations, and human exploration, space exploration has transformed our understanding of the universe and pushed the boundaries of human achievement. As we look to the future, space exploration holds the promise of unlocking even more profound insights and shaping the destiny of our species in the vastness of the cosmos.
+Space exploration continues to inspire and ignite our collective imagination. It is a testament to humanity's insatiable curiosity and our desire to unravel the mysteries of the cosmos. Through scientific discoveries, technological innovations, and human exploration, space exploration has transformed our understanding of the universe and pushed the boundaries of human achievement. As we look to the future, space exploration holds the promise of unlocking even more profound insights and shaping the destiny of our species in the vastness of the cosmos.
+
+A 34-year-old female, presented to the emergency room with severe abdominal pain and vomiting. She described the pain as constant and sharp, primarily located in the lower right quadrant of her abdomen. She had been experiencing these symptoms throughout the day, along with multiple episodes of non-bloody, non-bilious vomiting and persistent nausea. Jane also had a low-grade fever of 100.4°F (38°C). Physical examination revealed localized tenderness and guarding in the right lower quadrant of her abdomen. Diagnostic tests, including a complete blood count, urinalysis, and abdominal ultrasound, were ordered to determine the cause of her symptoms. In the meantime, Jane received symptomatic relief with IV fluids, analgesics for pain management, and antiemetics to control nausea. Further interventions, such as surgical consultation or additional imaging, may be necessary based on the diagnostic results.
+
+Title: "Shadows of the Past"
+
+INT. POLICE STATION - DAY
+
+Detective LUCAS, a seasoned investigator, sits at his desk, poring over case files. His partner, Detective AMY, enters the room.
+
+AMY
+(Looking concerned)
+Lucas, have you heard about the new lead on the Thompson case?
+
+LUCAS
+(With intrigue)
+No, what's the update?
+
+Amy hands Lucas a folder containing photographs and documents.
+
+AMY
+Our team received an anonymous tip. It seems someone has information about the night of the murder.
+
+LUCAS
+(Examining the folder)
+This might be the break we've been waiting for. Let's track down this witness immediately.
+
+INT. APARTMENT - LIVING ROOM - DAY
+
+Lucas and Amy arrive at an apartment building. They knock on the door of APARTMENT 307. A weary-looking MAN opens the door.
+
+LUCAS
+(Showing his badge)
+Mr. Reynolds, we're from the police department. We received information that you might have knowledge about the Thompson murder.
+
+MR. REYNOLDS
+(Nervously)
+I... I don't know anything. You must be mistaken.
+
+AMY
+(Gently)
+Mr. Reynolds, we're just looking for the truth. We believe you might have been a witness that night. Please, help us find justice for the Thompson family.
+
+Mr. Reynolds hesitates for a moment, then reluctantly invites them into his apartment.
+
+INT. APARTMENT - MR. REYNOLDS' LIVING ROOM - DAY
+
+Lucas and Amy sit across from Mr. Reynolds, who seems visibly uncomfortable.
+
+LUCAS
+(Trying to reassure him)
+Mr. Reynolds, we understand you might be afraid, but we're here to protect you. We believe you can shed light on what happened that night.
+
+MR. REYNOLDS
+(Sighs)
+Fine. I saw something that night, but I've been too scared to come forward.
+
+He recounts the events of the night of the murder, describing a mysterious figure lurking near the crime scene.
+
+INT. POLICE STATION - INTERROGATION ROOM - DAY
+
+Lucas and Amy review the information provided by Mr. Reynolds. They cross-reference it with other evidence, mapping out potential suspects.
+
+AMY
+(Excited)
+Lucas, this matches the description of a known criminal, Victor Sullivan. He has a history of violence.
+
+LUCAS
+(Grimly)
+Let's bring him in for questioning. We need to find out if he's connected to the Thompson murder.
+
+INT. POLICE STATION - INTERROGATION ROOM - DAY
+
+Detectives Lucas and Amy interrogate Victor Sullivan, a hardened criminal with a menacing demeanor.
+
+LUCAS
+(Intense)
+Victor, we know about your involvement in various criminal activities. But today, we're here to talk about the Thompson murder.
+
+VICTOR SULLIVAN
+(Smirking)
+I don't know what you're talking about. I've got nothing to do with it.
+
+AMY
+(Leaning forward)
+Mr. Sullivan, we have an eyewitness who places you near the crime scene that night. Don't make it harder for yourself.
+
+Victor's smug expression falters, revealing a glimmer of concern.
+
+INT. POLICE STATION - OBSERVATION ROOM - DAY
+
+As the detectives watch the interrogation through a one-way mirror, they exchange knowing glances.
+
+LUCAS
+(Whispering)
+We've got him on the ropes. Let's keep the pressure on and see if he cracks.
+
+AMY
+(Assured)
+We're close, Lucas. We'll get justice for the Thompsons.
+
+The tension builds as they continue the intense interrogation.
+
+Hey there, my name is Alex and I'm a 27-year-old guy living in a bustling city. I've always been drawn to the energy and diversity of urban life. I was born and raised in this city, and I feel a deep connection to its vibrant culture and constant buzz.
+
+By profession, I'm a graphic designer with a passion for visual storytelling. I love bringing ideas to life through colors, shapes, and typography. Whether it's designing a logo, creating a captivating website, or crafting eye-catching illustrations, I find immense joy in the creative process.
+
+Outside of work, I have a wide range of interests that keep me engaged and fulfilled. I'm an avid traveler and an explorer at heart. I've been fortunate enough to venture to various corners of the world, immersing myself in different cultures and embracing new experiences. The thrill of discovering unfamiliar landscapes and connecting with people from different walks of life is something I cherish deeply.
+
+Nature has a special place in my heart. I find solace and inspiration in the great outdoors. Hiking through lush forests, capturing breathtaking sunsets, and breathing in the crisp mountain air recharges my soul. It's a beautiful reminder of the wonders that exist beyond the concrete jungle.
+
+When I'm not busy with work or satisfying my wanderlust, I enjoy spending quality time with friends and loved ones. Whether it's trying out new restaurants, catching a movie, or engaging in deep conversations over a cup of coffee, the connections I have with the people around me are incredibly important.
+
+In terms of personal growth, I'm constantly seeking new knowledge and expanding my horizons. Books are my companions on this journey, allowing me to delve into different realms of imagination and gain fresh perspectives. I also have a penchant for learning about history and philosophy, as they provide valuable insights into the complexities of the world.
+
+As I embrace each day with optimism and an open mind, I strive to make a positive impact on the world around me. Whether through my creative endeavors, my interactions with others, or my commitment to personal growth, I believe that small actions can have a ripple effect, contributing to a better future.
+
+That's a glimpse into who I am—a curious, creative, and compassionate individual with a zest for life and a desire to make a difference.
+
+Patient Narrative: A Visit to the Doctor
+
+I woke up this morning feeling a bit under the weather. As the day progressed, my symptoms seemed to worsen, leaving me feeling tired and achy. Concerned about my health, I decided to schedule an appointment with my trusted family doctor, Dr. Smith.
+
+Arriving at the clinic, I was greeted by the friendly receptionist who checked me in and handed me a clipboard with some paperwork to fill out. While sitting in the waiting area, I couldn't help but notice the pleasant atmosphere with soothing music playing softly in the background.
+
+After a short wait, Nurse Johnson called my name and guided me to the examination room. She took my vitals, including my temperature, blood pressure, and heart rate, all of which seemed within the normal range. Nurse Johnson was kind and made sure I felt comfortable throughout the process.
+
+A few minutes later, Dr. Smith walked into the room with a warm smile on her face. I've been seeing her for years, and her caring demeanor always put me at ease. She asked how I was doing and listened attentively as I described my symptoms: persistent headache, body aches, and a sore throat.
+
+Dr. Smith took a moment to review my medical history and asked if I had any recent travel or exposure to sick individuals. She then proceeded to perform a thorough physical examination, carefully checking my throat, lymph nodes, and listening to my lungs and heart. Throughout the examination, she explained what she was doing and ensured that I understood each step.
+
+Once the examination was complete, Dr. Smith discussed her findings with me. She suspected that I might have caught a common viral infection, but to rule out any other possibilities, she suggested running a few tests. She explained the tests in detail, including how they would help in making an accurate diagnosis.
+
+Understanding the importance of communication, Dr. Smith encouraged me to ask questions and provided clear explanations. I appreciated her patience and felt reassured that I was in good hands.
+
+After discussing the treatment options, we decided that a combination of rest, over-the-counter pain relievers, and plenty of fluids would be the initial approach. However, Dr. Smith assured me that if my symptoms persisted or worsened, I should contact the clinic to arrange a follow-up visit.
+
+Before leaving, Dr. Smith kindly provided me with a written summary of our discussion, including instructions on managing my symptoms and a prescription for pain relief medication, just in case. She emphasized the importance of self-care and advised me to monitor my condition closely.
+
+Feeling grateful for the care I received, I thanked Dr. Smith and her team for their professionalism and compassion. As I left the clinic, I felt confident that I had taken the necessary steps to address my health concerns and that Dr. Smith would continue to support me on my journey to recovery.
+
+Overall, my visit to the doctor was a positive experience, reaffirming my trust in the healthcare system and the expertise of my physician.
+
+Project Name: ChatBotPlus
+ChatBotPlus is an open-source project that aims to provide a versatile and customizable chatbot framework. This project leverages the power of natural language processing and machine learning to create intelligent conversational agents that can be integrated into various applications.
+
+Features
+Natural Language Processing: ChatBotPlus utilizes advanced natural language processing techniques to understand and interpret user input, enabling it to engage in meaningful conversations.
+Machine Learning: The chatbot framework is powered by machine learning algorithms that enable the bot to learn and improve over time, providing more accurate and relevant responses.
+Customizable Responses: Developers can easily customize the bot's responses to suit specific requirements and tailor the chatbot's personality to match the desired user experience.
+Multi-platform Support: ChatBotPlus is designed to be platform-agnostic, allowing integration with a wide range of applications, including websites, mobile apps, and messaging platforms.
+Extensible Architecture: The project offers a modular and extensible architecture, allowing developers to add new features, integrate with external APIs, and enhance the functionality of the chatbot.
+
+a 45-year-old male, was admitted to the emergency room (ER) with complaints of severe chest pain and shortness of breath.
+
+History of Present Illness:
+According to the patient, he was at home when he suddenly experienced a crushing pain in his chest that radiated down his left arm. He also complained of feeling lightheaded and having difficulty breathing. Recognizing these symptoms as potential signs of a heart attack, he immediately called emergency services, and an ambulance brought him to the ER.
+
+Past Medical History:
+Mr. Smith has a history of hypertension, for which he takes medication irregularly. He has no known allergies. He denies any previous cardiac events or surgeries.
+
+Social History:
+The patient is a non-smoker and reports consuming alcohol occasionally on social occasions. He works as an accountant and denies any recent travel or exposure to sick individuals. He lives alone but has two adult children who live nearby.
+
+Family History:
+His father had a history of heart disease and suffered a myocardial infarction at the age of 55. His mother is alive and has no significant medical conditions. He has one sibling who is healthy.
+
+Review of Systems:
+Apart from the chief complaints, the patient denies any other specific symptoms. He does not report any recent fever, cough, abdominal pain, or changes in bowel or bladder habits.
+
+Physical Examination:
+Upon arrival at the ER, Mr. Smith appeared anxious and was diaphoretic. His blood pressure was elevated at 160/95 mmHg, heart rate was 110 beats per minute, respiratory rate was 24 breaths per minute, and oxygen saturation was 94% on room air. On auscultation, the lungs were clear bilaterally. Cardiac examination revealed regular rhythm with no murmurs, gallops, or rubs. The abdomen was soft and non-tender.
+
+Diagnostic Evaluation:
+Upon admission to the ER, the following tests were ordered:
+
+1. Electrocardiogram (ECG): Showed ST-segment elevation in leads II, III, and aVF, suggestive of an inferior wall myocardial infarction.
+
+2. Cardiac enzymes: Troponin I levels were elevated, indicating myocardial damage.
+
+3. Chest X-ray: No acute abnormalities noted.
+
+Assessment:
+Based on the patient's clinical presentation, elevated cardiac enzymes, and ECG findings, the initial diagnosis is an acute myocardial infarction, specifically involving the inferior wall of the heart.
+
+Plan:
+The patient was promptly started on supplemental oxygen via nasal cannula to maintain oxygen saturation above 94%. Intravenous access was established, and he was given aspirin and nitroglycerin for symptom relief. Sublingual nitroglycerin was administered with subsequent improvement in chest pain. An order was placed for urgent coronary angiography and possible percutaneous coronary intervention (PCI) if appropriate.
+
+The patient was admitted to the cardiac care unit for close monitoring. His blood pressure was managed with antihypertensive medication, and he was started on dual antiplatelet therapy (aspirin and clopidogrel). He was also given heparin to prevent clot formation.
+
+The patient's family was notified about the situation, and they arrived at the hospital to provide support. The medical team explained the diagnosis, treatment plan, and the potential need for PCI. The patient was encouraged to make lifestyle modifications, including smoking cessation, regular exercise, and adherence to medications.
+
+The case was discussed with the interventional cardiology team, who planned to perform coronary angiography and evaluate the suitability for PCI within the next 24 hours.
+
+Ongoing monitoring and close observation were implemented to ensure early detection
+
+Patient Chart
+
+Patient Information:
+Name: Emily Johnson
+Date of Birth: March 12, 1985
+Gender: Female
+Address: 1234 Maple Street, Anytown, USA
+Contact Number: (555) 123-4567
+Emergency Contact: Sarah Johnson
+Emergency Contact Number: (555) 987-6543
+
+Medical History:
+- Allergies: None known
+- Past Medical History: Appendectomy in 2007, Hypothyroidism
+- Medications: Levothyroxine 100mcg once daily
+
+Presenting Complaint:
+The patient presents with a persistent cough and shortness of breath.
+
+Vital Signs:
+- Temperature: 98.6°F
+- Blood Pressure: 120/80 mmHg
+- Pulse Rate: 80 beats per minute
+- Respiratory Rate: 18 breaths per minute
+- Oxygen Saturation: 97%
+
+Physical Examination:
+General appearance: Well-nourished, alert, and oriented.
+Head and neck examination: No signs of swelling or abnormalities.
+Cardiovascular examination: Regular rate and rhythm, no murmurs or abnormal sounds.
+Respiratory examination: Decreased breath sounds and crackles in the lower lung fields.
+Abdominal examination: Soft, non-tender, and non-distended abdomen.
+Musculoskeletal examination: Full range of motion, no deformities or tenderness.
+
+Diagnostic Tests:
+Chest X-ray: Patchy infiltrates in the lower lobes consistent with pneumonia.
+
+Assessment and Plan:
+Assessment: Community-acquired pneumonia
+Plan:
+1. Prescribe oral antibiotics (Azithromycin 500mg once daily for 5 days).
+2. Recommend rest, increased fluid intake, and symptomatic relief with over-the-counter cough medicine.
+3. Advise follow-up appointment in 1 week for reassessment.
+
+Discussion and Counseling:
+Discussed the diagnosis of community-acquired pneumonia with the patient. Explained the importance of completing the full course of antibiotics and the expected timeframe for symptom improvement. Emphasized the need for rest and increased fluid intake. Provided information on over-the-counter cough medicine for symptomatic relief. Informed the patient about the possibility of contagiousness and advised her to practice good respiratory hygiene.
+
+Follow-up Instructions:
+1. Take Azithromycin 500mg tablet once daily for 5 days.
+2. Rest at home and avoid strenuous activities until symptoms improve.
+3. Increase fluid intake to maintain hydration.
+4. Take over-the-counter cough medicine as needed for symptom relief.
+5. Schedule a follow-up appointment on May 17, 2023, at 10:00 AM.
+6. If symptoms worsen or difficulty breathing occurs, seek immediate medical attention.
+
+Signature:
+Dr. Jessica Adams
+May 10, 2023, 11:30 AM

samples/human-generated/cdc_covid.txt

@@ -0,0 +1,5 @@
+WHO begins using the phrase “2019 Novel Coronavirus” or “2019-nCoV” to refer to disease causing the outbreak in Wuhan, China.
+
+CDC publishes information about the 2019 Novel Coronavirus (2019-nCoV) outbreak caused by the SARS CoV-2 virus on its website.
+
+Edward C. Holmes of the University of Sydney, Australia posts online that the viral genome sequence of the unknown pneumonia causing the outbreak in Wuhan has been uploaded to GenBank as “Wuhan-Hu-1” (MN908947) and will be released shortly. He does so on behalf of Yong-Zhen Zhang of Fudan University, Shanghai in collaboration with the Shanghai School of Public Health, the Central Hospital of Wuhan, Huazhong University of Science and Technology, the Wuhan Center for Disease Control and Prevention, the National Institute for Communicable Disease Control and Prevention, Chinese Center for Disease Control, and the University of Sydney. Hours later, Holmes and Zhang publish the sequence.

samples/human-generated/cdc_covid_asthma.txt

@@ -0,0 +1,33 @@
+Follow your Asthma Action Plan
+Keep your asthma under control by following your asthma action plan.
+Avoid your asthma triggers.
+Continue current medications, including any inhalers with steroids in them (“steroids” is another word for corticosteroids). Know how to use your inhaler.
+Do not stop any medications or change your asthma treatment plan without talking to your healthcare provider.
+Talk to your healthcare provider, insurer, and pharmacist about creating an emergency supply of prescription medications, such as asthma inhalers. Make sure that you have 30 days of non-prescription medications and supplies on hand in case you need to stay home for a long time.
+Be careful around cleaning agents and disinfectants
+Follow the recommendations below to reduce your chance of an asthma attack while cleaning. Follow recommendations for cleaning your home and in your facility.
+Cleaning and Disinfecting Your Home
+
+If you have asthma:
+Ask an adult without asthma to clean and disinfect surfaces and objects for you.
+Stay in another room when cleaners or disinfectants are being used and right after their use.
+Use cleaning agents and disinfectant only when necessary. In routine situations, high-touch surfaces and objects might be cleaned effectively with soap and water.
+Make a list of the urgent care or health facilities near you that provides nebulizer/asthma treatments and keep it close to your phone.
+If you have an asthma attack, move away from the trigger, such as the cleaning agent or disinfectant or the area that was disinfected. Follow your Asthma Action Plan. Call 911 for medical emergencies.
+The person cleaning and disinfecting should:
+Choose disinfectants that are less likely to cause an asthma attack, using the U.S. Environmental Protection Agency (EPA)’s list of approved products, such as:
+Products with hydrogen peroxide (no stronger than 3%) or ethanol (ethyl alcohol). Ensure that products with hydrogen peroxide do not contain other chemicals that can trigger asthma attacks such as peroxyacetic acid or peracetic acid.
+Limit use of chemicals that can trigger asthma attacks, such as bleach (sodium hypochlorite) or quaternary ammonium compounds (for example, benzalkonium chloride), and do not use them in enclosed spaces.
+Follow additional precautions for cleaning and disinfecting places where people with asthma might be, to reduce exposure to asthma triggers.
+Use products safely and correctly:
+Always read and follow the directions on the product label to ensure you are using it safely and effectively.
+Wear skin protection such as gloves and consider eye protection to protect yourself against splashes.
+Make sure there is enough air flow (ventilation).
+Use only the amount recommended on the label.
+Use water at room temperature for dilution (unless stated otherwise on the label).
+Do NOT mix chemical products. Label diluted cleaning solutions.
+Spray or pour spray products onto a cleaning cloth or paper towel instead of spraying the product directly onto the cleaning surface (if the product label allows).
+Store products safely and correctly
+Store and use chemicals out of the reach of children and pets.
+Label diluted cleaning solutions.
+Follow EPA’s 6 steps for Safe and Effective Disinfectant Use.

samples/human-generated/lzma_readme.txt

@@ -1,6 +1,5 @@
-This is the first attempt, using the LZMA compression ratios as a way to indirectly measure the perplexity of a text. Compression ratios have been used in the past to detect anomalies 
-in network data for intrusion detection, so if perplexity is roughly a measure of anomalous tokens, it may be possible to use compression to detect low-perplexity text. LZMA creates 
-a dictionary of seen tokens, and then uses though in place of future tokens. The dictionary size, token length, etc. are all dynamic (though influenced by the 'preset' of 0-9--with 0 
-being the fastest but worse compression than 9). The basic idea is to 'seed' an LZMA compression stream with a corpus of AI-generated text (ai-generated.txt) and then measure the 
-compression ratio of just the seed data with that of the sample appended. Samples that follow more closely in word choice, structure, etc. will acheive a higher compression ratio due 
-to the prevalence of similar tokens in the dictionary, novel words, structures, etc. will appear anomalous to the seeded dictionary, resulting in a worse compression ratio.
+ai-detect: Fast methods to classify text as AI or human-generated
+This is a research repo for fast AI detection methods as we experiment with different techniques. While there are a number of existing LLM detection systems, they all use a large model trained on either an LLM or its training data to calculate the probability of each word given the preceeding, then calculating a score where the more high-probability tokens are more likely to be AI-originated. Techniques and tools in this repo are looking for faster approximation to be embeddable and more scalable.
+
+LZMA compression detector (lzma_detect.py)
+This is the first attempt, using the LZMA compression ratios as a way to indirectly measure the perplexity of a text. Compression ratios have been used in the past to detect anomalies in network data for intrusion detection, so if perplexity is roughly a measure of anomalous tokens, it may be possible to use compression to detect low-perplexity text. LZMA creates a dictionary of seen tokens, and then uses though in place of future tokens. The dictionary size, token length, etc. are all dynamic (though influenced by the 'preset' of 0-9--with 0 being the fastest but worse compression than 9). The basic idea is to 'seed' an LZMA compression stream with a corpus of AI-generated text (ai-generated.txt) and then measure the compression ratio of just the seed data with that of the sample appended. Samples that follow more closely in word choice, structure, etc. will acheive a higher compression ratio due to the prevalence of similar tokens in the dictionary, novel words, structures, etc. will appear anomalous to the seeded dictionary, resulting in a worse compression ratio.

samples/human-generated/pg_robinson_crusoe.txt

@@ -0,0 +1,55 @@
+I was born in the year 1632, in the city of York, of a good family, though not of that country, my father being a foreigner of Bremen, who settled first at Hull. He got a good estate by merchandise, and leaving off his trade, lived afterwards at York, from whence he had married my mother, whose relations were named Robinson, a very good family in that country, and from whom I was called Robinson Kreutznaer; but, by the usual corruption of words in England, we are now called—nay we call ourselves and write our name—Crusoe; and so my companions always called me.
+
+I had two elder brothers, one of whom was lieutenant-colonel to an English regiment of foot in Flanders, formerly commanded by the famous Colonel Lockhart, and was killed at the battle near Dunkirk against the Spaniards. What became of my second brother I never knew, any more than my father or mother knew what became of me.
+
+Being the third son of the family and not bred to any trade, my head began to be filled very early with rambling thoughts. My father, who was very ancient, had given me a competent share of learning, as far as house-education and a country free school generally go, and designed me for the law; but I would be satisfied with nothing but going to sea; and my inclination to this led me so strongly against the will, nay, the commands of my father, and against all the entreaties and persuasions of my mother and other friends, that there seemed to be something fatal in that propensity of nature, tending directly to the life of misery which was to befall me.
+
+My father, a wise and grave man, gave me serious and excellent counsel against what he foresaw was my design. He called me one morning into his chamber, where he was confined by the gout, and expostulated very warmly with me upon this subject. He asked me what reasons, more than a mere wandering inclination, I had for leaving father’s house and my native country, where I might be well introduced, and had a prospect of raising my fortune by application and industry, with a life of ease and pleasure. He told me it was men of desperate fortunes on one hand, or of aspiring, superior fortunes on the other, who went abroad upon adventures, to rise by enterprise, and make themselves famous in undertakings of a nature out of the common road; that these things were all either too far above me or too far below me; that mine was the middle state, or what might be called the upper station of low life, which he had found, by long experience, was the best state in the world, the most suited to human happiness, not exposed to the miseries and hardships, the labour and sufferings of the mechanic part of mankind, and not embarrassed with the pride, luxury, ambition, and envy of the upper part of mankind. He told me I might judge of the happiness of this state by this one thing—viz. that this was the state of life which all other people envied; that kings have frequently lamented the miserable consequence of being born to great things, and wished they had been placed in the middle of the two extremes, between the mean and the great; that the wise man gave his testimony to this, as the standard of felicity, when he prayed to have neither poverty nor riches.
+
+He bade me observe it, and I should always find that the calamities of life were shared among the upper and lower part of mankind, but that the middle station had the fewest disasters, and was not exposed to so many vicissitudes as the higher or lower part of mankind; nay, they were not subjected to so many distempers and uneasinesses, either of body or mind, as those were who, by vicious living, luxury, and extravagances on the one hand, or by hard labour, want of necessaries, and mean or insufficient diet on the other hand, bring distemper upon themselves by the natural consequences of their way of living; that the middle station of life was calculated for all kind of virtue and all kind of enjoyments; that peace and plenty were the handmaids of a middle fortune; that temperance, moderation, quietness, health, society, all agreeable diversions, and all desirable pleasures, were the blessings attending the middle station of life; that this way men went silently and smoothly through the world, and comfortably out of it, not embarrassed with the labours of the hands or of the head, not sold to a life of slavery for daily bread, nor harassed with perplexed circumstances, which rob the soul of peace and the body of rest, nor enraged with the passion of envy, or the secret burning lust of ambition for great things; but, in easy circumstances, sliding gently through the world, and sensibly tasting the sweets of living, without the bitter; feeling that they are happy, and learning by every day’s experience to know it more sensibly.
+
+After this he pressed me earnestly, and in the most affectionate manner, not to play the young man, nor to precipitate myself into miseries which nature, and the station of life I was born in, seemed to have provided against; that I was under no necessity of seeking my bread; that he would do well for me, and endeavour to enter me fairly into the station of life which he had just been recommending to me; and that if I was not very easy and happy in the world, it must be my mere fate or fault that must hinder it; and that he should have nothing to answer for, having thus discharged his duty in warning me against measures which he knew would be to my hurt; in a word, that as he would do very kind things for me if I would stay and settle at home as he directed, so he would not have so much hand in my misfortunes as to give me any encouragement to go away; and to close all, he told me I had my elder brother for an example, to whom he had used the same earnest persuasions to keep him from going into the Low Country wars, but could not prevail, his young desires prompting him to run into the army, where he was killed; and though he said he would not cease to pray for me, yet he would venture to say to me, that if I did take this foolish step, God would not bless me, and I should have leisure hereafter to reflect upon having neglected his counsel when there might be none to assist in my recovery.
+
+I observed in this last part of his discourse, which was truly prophetic, though I suppose my father did not know it to be so himself—I say, I observed the tears run down his face very plentifully, especially when he spoke of my brother who was killed: and that when he spoke of my having leisure to repent, and none to assist me, he was so moved that he broke off the discourse, and told me his heart was so full he could say no more to me.
+
+I was sincerely affected with this discourse, and, indeed, who could be otherwise? and I resolved not to think of going abroad any more, but to settle at home according to my father’s desire. But alas! a few days wore it all off; and, in short, to prevent any of my father’s further importunities, in a few weeks after I resolved to run quite away from him. However, I did not act quite so hastily as the first heat of my resolution prompted; but I took my mother at a time when I thought her a little more pleasant than ordinary, and told her that my thoughts were so entirely bent upon seeing the world that I should never settle to anything with resolution enough to go through with it, and my father had better give me his consent than force me to go without it; that I was now eighteen years old, which was too late to go apprentice to a trade or clerk to an attorney; that I was sure if I did I should never serve out my time, but I should certainly run away from my master before my time was out, and go to sea; and if she would speak to my father to let me go one voyage abroad, if I came home again, and did not like it, I would go no more; and I would promise, by a double diligence, to recover the time that I had lost.
+
+This put my mother into a great passion; she told me she knew it would be to no purpose to speak to my father upon any such subject; that he knew too well what was my interest to give his consent to anything so much for my hurt; and that she wondered how I could think of any such thing after the discourse I had had with my father, and such kind and tender expressions as she knew my father had used to me; and that, in short, if I would ruin myself, there was no help for me; but I might depend I should never have their consent to it; that for her part she would not have so much hand in my destruction; and I should never have it to say that my mother was willing when my father was not.
+
+Though my mother refused to move it to my father, yet I heard afterwards that she reported all the discourse to him, and that my father, after showing a great concern at it, said to her, with a sigh, “That boy might be happy if he would stay at home; but if he goes abroad, he will be the most miserable wretch that ever was born: I can give no consent to it.”
+
+It was not till almost a year after this that I broke loose, though, in the meantime, I continued obstinately deaf to all proposals of settling to business, and frequently expostulated with my father and mother about their being so positively determined against what they knew my inclinations prompted me to. But being one day at Hull, where I went casually, and without any purpose of making an elopement at that time; but, I say, being there, and one of my companions being about to sail to London in his father’s ship, and prompting me to go with them with the common allurement of seafaring men, that it should cost me nothing for my passage, I consulted neither father nor mother any more, nor so much as sent them word of it; but leaving them to hear of it as they might, without asking God’s blessing or my father’s, without any consideration of circumstances or consequences, and in an ill hour, God knows, on the 1st of September 1651, I went on board a ship bound for London. Never any young adventurer’s misfortunes, I believe, began sooner, or continued longer than mine. The ship was no sooner out of the Humber than the wind began to blow and the sea to rise in a most frightful manner; and, as I had never been at sea before, I was most inexpressibly sick in body and terrified in mind. I began now seriously to reflect upon what I had done, and how justly I was overtaken by the judgment of Heaven for my wicked leaving my father’s house, and abandoning my duty. All the good counsels of my parents, my father’s tears and my mother’s entreaties, came now fresh into my mind; and my conscience, which was not yet come to the pitch of hardness to which it has since, reproached me with the contempt of advice, and the breach of my duty to God and my father.
+
+All this while the storm increased, and the sea went very high, though nothing like what I have seen many times since; no, nor what I saw a few days after; but it was enough to affect me then, who was but a young sailor, and had never known anything of the matter. I expected every wave would have swallowed us up, and that every time the ship fell down, as I thought it did, in the trough or hollow of the sea, we should never rise more; in this agony of mind, I made many vows and resolutions that if it would please God to spare my life in this one voyage, if ever I got once my foot upon dry land again, I would go directly home to my father, and never set it into a ship again while I lived; that I would take his advice, and never run myself into such miseries as these any more. Now I saw plainly the goodness of his observations about the middle station of life, how easy, how comfortably he had lived all his days, and never had been exposed to tempests at sea or troubles on shore; and I resolved that I would, like a true repenting prodigal, go home to my father.
+
+These wise and sober thoughts continued all the while the storm lasted, and indeed some time after; but the next day the wind was abated, and the sea calmer, and I began to be a little inured to it; however, I was very grave for all that day, being also a little sea-sick still; but towards night the weather cleared up, the wind was quite over, and a charming fine evening followed; the sun went down perfectly clear, and rose so the next morning; and having little or no wind, and a smooth sea, the sun shining upon it, the sight was, as I thought, the most delightful that ever I saw.
+
+I had slept well in the night, and was now no more sea-sick, but very cheerful, looking with wonder upon the sea that was so rough and terrible the day before, and could be so calm and so pleasant in so little a time after. And now, lest my good resolutions should continue, my companion, who had enticed me away, comes to me; “Well, Bob,” says he, clapping me upon the shoulder, “how do you do after it? I warrant you were frighted, wer’n’t you, last night, when it blew but a capful of wind?” “A capful d’you call it?” said I; “’twas a terrible storm.” “A storm, you fool you,” replies he; “do you call that a storm? why, it was nothing at all; give us but a good ship and sea-room, and we think nothing of such a squall of wind as that; but you’re but a fresh-water sailor, Bob. Come, let us make a bowl of punch, and we’ll forget all that; d’ye see what charming weather ’tis now?” To make short this sad part of my story, we went the way of all sailors; the punch was made and I was made half drunk with it: and in that one night’s wickedness I drowned all my repentance, all my reflections upon my past conduct, all my resolutions for the future. In a word, as the sea was returned to its smoothness of surface and settled calmness by the abatement of that storm, so the hurry of my thoughts being over, my fears and apprehensions of being swallowed up by the sea being forgotten, and the current of my former desires returned, I entirely forgot the vows and promises that I made in my distress. I found, indeed, some intervals of reflection; and the serious thoughts did, as it were, endeavour to return again sometimes; but I shook them off, and roused myself from them as it were from a distemper, and applying myself to drinking and company, soon mastered the return of those fits—for so I called them; and I had in five or six days got as complete a victory over conscience as any young fellow that resolved not to be troubled with it could desire. But I was to have another trial for it still; and Providence, as in such cases generally it does, resolved to leave me entirely without excuse; for if I would not take this for a deliverance, the next was to be such a one as the worst and most hardened wretch among us would confess both the danger and the mercy of.
+
+The sixth day of our being at sea we came into Yarmouth Roads; the wind having been contrary and the weather calm, we had made but little way since the storm. Here we were obliged to come to an anchor, and here we lay, the wind continuing contrary—viz. at south-west—for seven or eight days, during which time a great many ships from Newcastle came into the same Roads, as the common harbour where the ships might wait for a wind for the river.
+
+We had not, however, rid here so long but we should have tided it up the river, but that the wind blew too fresh, and after we had lain four or five days, blew very hard. However, the Roads being reckoned as good as a harbour, the anchorage good, and our ground-tackle very strong, our men were unconcerned, and not in the least apprehensive of danger, but spent the time in rest and mirth, after the manner of the sea; but the eighth day, in the morning, the wind increased, and we had all hands at work to strike our topmasts, and make everything snug and close, that the ship might ride as easy as possible. By noon the sea went very high indeed, and our ship rode forecastle in, shipped several seas, and we thought once or twice our anchor had come home; upon which our master ordered out the sheet-anchor, so that we rode with two anchors ahead, and the cables veered out to the bitter end.
+
+By this time it blew a terrible storm indeed; and now I began to see terror and amazement in the faces even of the seamen themselves. The master, though vigilant in the business of preserving the ship, yet as he went in and out of his cabin by me, I could hear him softly to himself say, several times, “Lord be merciful to us! we shall be all lost! we shall be all undone!” and the like. During these first hurries I was stupid, lying still in my cabin, which was in the steerage, and cannot describe my temper: I could ill resume the first penitence which I had so apparently trampled upon and hardened myself against: I thought the bitterness of death had been past, and that this would be nothing like the first; but when the master himself came by me, as I said just now, and said we should be all lost, I was dreadfully frighted. I got up out of my cabin and looked out; but such a dismal sight I never saw: the sea ran mountains high, and broke upon us every three or four minutes; when I could look about, I could see nothing but distress round us; two ships that rode near us, we found, had cut their masts by the board, being deep laden; and our men cried out that a ship which rode about a mile ahead of us was foundered. Two more ships, being driven from their anchors, were run out of the Roads to sea, at all adventures, and that with not a mast standing. The light ships fared the best, as not so much labouring in the sea; but two or three of them drove, and came close by us, running away with only their spritsail out before the wind.
+
+Towards evening the mate and boatswain begged the master of our ship to let them cut away the fore-mast, which he was very unwilling to do; but the boatswain protesting to him that if he did not the ship would founder, he consented; and when they had cut away the fore-mast, the main-mast stood so loose, and shook the ship so much, they were obliged to cut that away also, and make a clear deck.
+
+Any one may judge what a condition I must be in at all this, who was but a young sailor, and who had been in such a fright before at but a little. But if I can express at this distance the thoughts I had about me at that time, I was in tenfold more horror of mind upon account of my former convictions, and the having returned from them to the resolutions I had wickedly taken at first, than I was at death itself; and these, added to the terror of the storm, put me into such a condition that I can by no words describe it. But the worst was not come yet; the storm continued with such fury that the seamen themselves acknowledged they had never seen a worse. We had a good ship, but she was deep laden, and wallowed in the sea, so that the seamen every now and then cried out she would founder. It was my advantage in one respect, that I did not know what they meant by founder till I inquired. However, the storm was so violent that I saw, what is not often seen, the master, the boatswain, and some others more sensible than the rest, at their prayers, and expecting every moment when the ship would go to the bottom. In the middle of the night, and under all the rest of our distresses, one of the men that had been down to see cried out we had sprung a leak; another said there was four feet water in the hold. Then all hands were called to the pump. At that word, my heart, as I thought, died within me: and I fell backwards upon the side of my bed where I sat, into the cabin. However, the men roused me, and told me that I, that was able to do nothing before, was as well able to pump as another; at which I stirred up and went to the pump, and worked very heartily. While this was doing the master, seeing some light colliers, who, not able to ride out the storm were obliged to slip and run away to sea, and would come near us, ordered to fire a gun as a signal of distress. I, who knew nothing what they meant, thought the ship had broken, or some dreadful thing happened. In a word, I was so surprised that I fell down in a swoon. As this was a time when everybody had his own life to think of, nobody minded me, or what was become of me; but another man stepped up to the pump, and thrusting me aside with his foot, let me lie, thinking I had been dead; and it was a great while before I came to myself.
+
+We worked on; but the water increasing in the hold, it was apparent that the ship would founder; and though the storm began to abate a little, yet it was not possible she could swim till we might run into any port; so the master continued firing guns for help; and a light ship, who had rid it out just ahead of us, ventured a boat out to help us. It was with the utmost hazard the boat came near us; but it was impossible for us to get on board, or for the boat to lie near the ship’s side, till at last the men rowing very heartily, and venturing their lives to save ours, our men cast them a rope over the stern with a buoy to it, and then veered it out a great length, which they, after much labour and hazard, took hold of, and we hauled them close under our stern, and got all into their boat. It was to no purpose for them or us, after we were in the boat, to think of reaching their own ship; so all agreed to let her drive, and only to pull her in towards shore as much as we could; and our master promised them, that if the boat was staved upon shore, he would make it good to their master: so partly rowing and partly driving, our boat went away to the northward, sloping towards the shore almost as far as Winterton Ness.
+
+We were not much more than a quarter of an hour out of our ship till we saw her sink, and then I understood for the first time what was meant by a ship foundering in the sea. I must acknowledge I had hardly eyes to look up when the seamen told me she was sinking; for from the moment that they rather put me into the boat than that I might be said to go in, my heart was, as it were, dead within me, partly with fright, partly with horror of mind, and the thoughts of what was yet before me.
+
+While we were in this condition—the men yet labouring at the oar to bring the boat near the shore—we could see (when, our boat mounting the waves, we were able to see the shore) a great many people running along the strand to assist us when we should come near; but we made but slow way towards the shore; nor were we able to reach the shore till, being past the lighthouse at Winterton, the shore falls off to the westward towards Cromer, and so the land broke off a little the violence of the wind. Here we got in, and though not without much difficulty, got all safe on shore, and walked afterwards on foot to Yarmouth, where, as unfortunate men, we were used with great humanity, as well by the magistrates of the town, who assigned us good quarters, as by particular merchants and owners of ships, and had money given us sufficient to carry us either to London or back to Hull as we thought fit.
+
+Had I now had the sense to have gone back to Hull, and have gone home, I had been happy, and my father, as in our blessed Saviour’s parable, had even killed the fatted calf for me; for hearing the ship I went away in was cast away in Yarmouth Roads, it was a great while before he had any assurances that I was not drowned.
+
+But my ill fate pushed me on now with an obstinacy that nothing could resist; and though I had several times loud calls from my reason and my more composed judgment to go home, yet I had no power to do it. I know not what to call this, nor will I urge that it is a secret overruling decree, that hurries us on to be the instruments of our own destruction, even though it be before us, and that we rush upon it with our eyes open. Certainly, nothing but some such decreed unavoidable misery, which it was impossible for me to escape, could have pushed me forward against the calm reasonings and persuasions of my most retired thoughts, and against two such visible instructions as I had met with in my first attempt.
+
+My comrade, who had helped to harden me before, and who was the master’s son, was now less forward than I. The first time he spoke to me after we were at Yarmouth, which was not till two or three days, for we were separated in the town to several quarters; I say, the first time he saw me, it appeared his tone was altered; and, looking very melancholy, and shaking his head, he asked me how I did, and telling his father who I was, and how I had come this voyage only for a trial, in order to go further abroad, his father, turning to me with a very grave and concerned tone “Young man,” says he, “you ought never to go to sea any more; you ought to take this for a plain and visible token that you are not to be a seafaring man.” “Why, sir,” said I, “will you go to sea no more?” “That is another case,” said he; “it is my calling, and therefore my duty; but as you made this voyage on trial, you see what a taste Heaven has given you of what you are to expect if you persist. Perhaps this has all befallen us on your account, like Jonah in the ship of Tarshish. Pray,” continues he, “what are you; and on what account did you go to sea?” Upon that I told him some of my story; at the end of which he burst out into a strange kind of passion: “What had I done,” says he, “that such an unhappy wretch should come into my ship? I would not set my foot in the same ship with thee again for a thousand pounds.” This indeed was, as I said, an excursion of his spirits, which were yet agitated by the sense of his loss, and was farther than he could have authority to go. However, he afterwards talked very gravely to me, exhorting me to go back to my father, and not tempt Providence to my ruin, telling me I might see a visible hand of Heaven against me. “And, young man,” said he, “depend upon it, if you do not go back, wherever you go, you will meet with nothing but disasters and disappointments, till your father’s words are fulfilled upon you.”
+
+We parted soon after; for I made him little answer, and I saw him no more; which way he went I knew not. As for me, having some money in my pocket, I travelled to London by land; and there, as well as on the road, had many struggles with myself what course of life I should take, and whether I should go home or to sea.
+
+As to going home, shame opposed the best motions that offered to my thoughts, and it immediately occurred to me how I should be laughed at among the neighbours, and should be ashamed to see, not my father and mother only, but even everybody else; from whence I have since often observed, how incongruous and irrational the common temper of mankind is, especially of youth, to that reason which ought to guide them in such cases—viz. that they are not ashamed to sin, and yet are ashamed to repent; not ashamed of the action for which they ought justly to be esteemed fools, but are ashamed of the returning, which only can make them be esteemed wise men.
+
+In this state of life, however, I remained some time, uncertain what measures to take, and what course of life to lead. An irresistible reluctance continued to going home; and as I stayed away a while, the remembrance of the distress I had been in wore off, and as that abated, the little motion I had in my desires to return wore off with it, till at last I quite laid aside the thoughts of it, and looked out for a voyage.

samples/llm-generated/gpt_patient_narrative.txt

@@ -0,0 +1,12 @@
+Patient is a 35-year-old female who presents to the clinic complaining of a persistent cough and shortness of breath for the past two weeks. The patient reports that the symptoms 
+have been worsening and she is having difficulty sleeping due to the cough. The patient also reports that she has lost her sense of smell and taste. The patient denies any fever, chills, 
+chest pain, or gastrointestinal symptoms.
+
+The patient's medical history is significant for asthma, for which she uses an albuterol inhaler as needed. She has no other known medical problems and takes no other medications.
+
+On physical examination, the patient appears comfortable and in no distress. Lung examination reveals clear breath sounds with minimal wheezing. The patient's oxygen saturation is 96% on room air. 
+The rest of the examination is within normal limits.
+
+Based on the patient's symptoms and history, the provider suspects that the patient may have a respiratory infection, such as COVID-19. The provider orders a COVID-19 PCR test and advises the 
+patient to self-quarantine until the test results are available. The provider also prescribes an antibiotic and a bronchodilator to help alleviate the patient's symptoms. The provider advises the 
+patient to follow up with the clinic if her symptoms worsen or if she develops any new symptoms.

samples/llm-generated/gpt_patient_summary.txt

@@ -0,0 +1,3 @@
+A 35-year-old female patient presents to the clinic with a persistent cough, shortness of breath, loss of sense of smell and taste for 2 weeks. The patient has a 
+history of asthma and is experiencing difficulty sleeping. The provider suspects the patient has a respiratory infection and orders a COVID-19 PCR test and advises self-quarantine. 
+The provider prescribes an antibiotic and bronchodilator to alleviate symptoms and advises the patient to follow up if symptoms worsen or new symptoms appear.

samples/llm-generated/gpt_santos_short.txt

@@ -1,3 +1,5 @@
-George Santos Charged with Federal Financial Crimes
-
-New York City, May 10, 2023 – In a significant development today, George Santos, a prominent business figure, has been charged with multiple federal financial crimes. The charges were filed by the United States Department of Justice following an extensive investigation into Santos' financial activities.
+In a significant development today, George Santos, a prominent business figure, has been charged with multiple federal financial crimes. The charges were 
+filed by the United States Department of Justice following an extensive investigation into Santos' financial activities.
+According to the indictment, Santos faces allegations of money laundering, wire fraud, and securities fraud. The charges stem from his alleged involvement 
+in a complex scheme that defrauded investors and manipulated financial markets for personal gain. Federal authorities have been closely examining Santos' financial 
+transactions and activities over an extended period.