Flux Handwriting Model: AI Mimicing Human Handwriting

I never pictured AI writing human-like text, no I’m not talking about the text generation but rather image generation with human handwriting. The Flux models made it easy to infer, generate and edit images. Today, in this article, we’ll be looking at one such model used for generating images with hand-written text. No that’s not all, we’ll also build a story telling application towards the end of the article. 

What are FLUX Models?

Flux models are generative models that are often associated with generating high-quality images, videos, or other content. These models are built using advanced neural networks like Stable Diffusion or Variational Autoencoders (VAEs). We’ll be focusing on a Flux model, namely the fofr/flux-handwriting model throughout the article.

Flux Handwriting Model

“fofr/flux-hndwriting” is a flux Lora fine-tuned to produce handwritten text, let’s look at various ways to use it to generate some images with handwritten text.

Hugging Face

You can quickly to the model page on the Hugging Face and use the ‘diffusers library’ or the ‘inference api’ to generate the images. 

Note: Remember that you should use HWRIT handwriting to trigger the image generation.

I prompted it to generate: ““HWRIT shaky messy handwriting stating “The sun will rise,” illegible, dark green ink on old water-damaged paper with visible mold marks.““

The generated image has the text in the same style I had mentioned in the prompt.

Let’s try the Inference API

Get your HuggingFace access token from here: Hugging Face Tokens

from huggingface_hub import InferenceClient
client = InferenceClient("fofr/flux-handwriting", token="hf_token")
# output
image = client.text_to_image('HWRIT scrawling messy handwriting saying "I\'m Iron Man", illegible, written with a HB pencil on a grainy paper')

Output

This looks pretty good and the model did not mess up with any characters too. 

Note: It takes a while for the image generation. 

Replicate

You can also choose to run this model on Replicate but it’ll cost you approximately $1 for 90 runs  or approximately $0.11/run, this can vary.

Story Telling Application

Let’s create an LLM application that first writes a story and then breaks into 7 pieces and then generates 7 hand-written images to aid the storytelling. We’ll then combine these images to complete the application.

We’ll be using Gemini models to generate the story and make prompts to generate images from flux-handwriting. First, let’s get on our hands on the Gemini API-key:

Simply click on Create API Key to get a new key to use Gemini models.

Installation

!pip install -q -U google-generativeai

To use the Gemini models.

Implementation

Configure and choose the model, I’ll be using the ‘Gemini-1.5-flash’ model.

import google.generativeai as genai
genai.configure(api_key=”API-Key”)
model = genai.GenerativeModel("gemini-1.5-flash")

Generating the story:

response = model.generate_content("Write a short and clear story in about 80 words, about a day in the life of a man named Cyan turning into a superhero.")
story = response.text
print(story)

Cyan woke to a throbbing headache, a strange symbol burning into his palm.
  That day, mundane tasks – grocery shopping, dog walking – felt amplified,
 his senses sharper. A speeding car careened towards a child; instinctively,
 Cyan reacted.  He moved faster than he thought possible, a blur of motion,
 saving the child.  The symbol glowed.  He was no longer just Cyan. He was
 something more.

Now split the story into 7 parts:

sentences = story.split(". ")
prompts = [f"{sentences[i]}." if i < len(sentences) else "" for i in range(7)]

Structure the 7 parts into prompts to request for a response:

handwriting_prompts = [
   f"HWRIT handwriting style for the text: '{prompt}' in a neat cursive writing in orange Ink and red paper background"
   for prompt in prompts if prompt.strip()
]

Function to generate the handwritten images:

(Get your hugging face token and make sure to check all the inference boxes while creating a token)

from huggingface_hub import InferenceClient
import time
client = InferenceClient("fofr/flux-handwriting", token="hf_token")
def handwriting_text(prompt):
 image = client.text_to_image(prompt)
 return image

Generating images with handwritten text:

handwritten_images = []
for prompt in handwriting_prompts:
   image = handwriting_text(prompt)
   handwritten_images.append(image)
   time.sleep(120)  # 2-minute delay

Note: The API request might throw an error on the lines of “Max requests
 total reached on image generation inference (3). Wait up to one minute
 before being able to process more Diffusion requests.”, Hence we’re adding a
 120 second sleep after each request in the for loop.

Generating the video using OpenCV: 

import cv2
import os
def create_video_from_images(image_list, output_video_path, fps=1):
   # Load the first image to get dimensions
   frame = cv2.imread(image_list[0])
   height, width, _ = frame.shape
   # Initialize the video writer
   fourcc = cv2.VideoWriter_fourcc(*'mp4v')
   video = cv2.VideoWriter(output_video_path, fourcc, fps, (width, height))
   # Write each image to the video
   for image_path in image_list:
       frame = cv2.imread(image_path)
       video.write(frame)
   # Release the video writer
   video.release()
# Save the images to disk and create a video
image_file_paths = []
for idx, image in enumerate(handwritten_images):
   file_path = f"handwritten_image_{idx}.png"
   image.save(file_path)
   image_file_paths.append(file_path)
# Combine images into a video
create_video_from_images(image_file_paths, "handwritten_story.mp4", fps=0.25)
print("Video created: handwritten_story.mp4")

We saved the images and then combined them into a video with a frame rate of 0.25 (1 frame per 4 seconds for readability). 

Output

Link to the video: handwritten_story.mp4

Note: The model struggles while generating images with more than 4-5 words per image so we need to restrict the text. 

One exercise you could try is to use an LLM working to make the prompts instead of splitting the story and using a common template, this will ensure the text limit and the style and background of the text can be tuned according to the text by the LLM. 

Conclusion

In conclusion, using Flux models such as “fofr/flux-handwriting” introduces new opportunities for crafting custom handwritten-style visuals. Whether creating standalone prompts or developing full storytelling solutions, these tools highlight AI’s ability to merge artistic creativity with practical applications. The storytelling feature exemplifies how effortlessly AI-generated visuals can blend into multimedia projects, driving forward inventive and captivating possibilities. 

Also if you are looking for a Generative AI course online then, explore: GenAI Pinnacle Program

Frequently Asked Questions

Mounish V

I'm a tech enthusiast, graduated from Vellore Institute of Technology. I'm working as a Data Science Trainee right now. I am very much interested in Deep Learning and Generative AI.