Hamel dives deep into how LLM frameworks like langchain , instructor , and guidance perform tasks like formatting the response in a valid JSON output. He intercepts the API calls from these Python libraries to shed some light on how many API calls (to OpenAI’s GPT services) they make and what prompt they use. I’ve always been skeptic of the usefulness of many of the LLM “wrapper” libraries, specially for larger and more serious projects, as they are fine for quick prototypes.

Hamel’s blog post makes it clear that you should not blindly trust any of the LLM libraries, because some of them are just using some (stupid) prompt “engineering” behind the scene to provide you with good looking output, and they’re performance is pretty much hit and miss (unless you’re able to view they’re prompts and to verify they’re not doing anything silly).

An example from his post, investigating the API calls for guardrails :

from pydantic import BaseModel, Field
from guardrails import Guard
import openai

class Pet(BaseModel):
    pet_type: str = Field(description="Species of pet")
    name: str = Field(description="a unique pet name")

prompt = """
    What kind of pet should I get and what should I name it?

    ${gr.complete_json_suffix_v2}
"""
guard = Guard.from_pydantic(output_class=Pet, prompt=prompt)

validated_output, *rest = guard(
    llm_api=openai.completions.create,
    engine="gpt-3.5-turbo-instruct"
)

print(f"{validated_output}")

## {
##    "pet_type": "dog",
##    "name": "Buddy

Not a valid JSON output!

What is happening here? How is this structured output and validation working? Looking at the mitmproxy UI, I can see that the above code resulted in two LLM API calls, the first one with this prompt:

W G O H - - - h i o N e a v u o L r ` ` ` t e t u Y e < < < n p < < t s l o k u s s p r a t i b i b t t t u e r r s j n e > r r t t e i t e d l i i > u n c o n n r e g n t o w g g n x a f a n m n i n n a m a e a p s a a p m = m e m m v l e ' e t X e e a e = b = M = = l s ' a ' s L " " i f r b h p n d o o ' a o t e a f o > z u h t m J ' < ' l a _ e S s s > d t t " O i f t < y N m o r s I d p d p r i t e e e o l m n r g s " s b e a g i e c c j t n t r d r e ( = f g i e i c X ' o a b s p t M t r n n e c t L w m a d s r i ( , o a m i o n - t e w t p n o J w = = h h t = S o ' " a e i " o O r u f t o a t N d p o i n h ) s p o s n = u e e " h f " n r p l r o o S i a o - f u r p q t i w c o l m e u e r e a r d a c e x s r s m t i t - e a I i e p t c ' t o s e i h a = n n t s a s " a o t e c m t f n n ' a e o a e s p p m c h l i i e e e e > i t t x t " s w ` s a ? t " s t l r a t = > i a r h > ` z c y e e t ) ` = , e { > t f x ' w r w p f ` o o h e o { - m e c o " w r t ' b o t e e : a r h d r d i t ' " s s h b e : " e e x d h a [ o k a m ' c e v p S < u y i l T i m o e R n e o r I t n f : o N e t n G g t e e a h ' O r n e } N d ` E n f ' a t i , m h e e e l ' = d S " t T i a i R n g n I d s N e J G x t S " o O T N W f e O o x i ' r t s , m r a a t e t c h t = t e c " . 1 i ` ] - t n } i a ` n i m d n e e t ` x o e . a d t " t r i b u t o e b j o e f c t t > h ` e = c > o r ` r { e ' s b p a o z n ' d : i n { g ' f X o M o L ' , : a ' n S d o m t e h e S t v r a i l n u g e ' , i s ' i o n f d e t x h ' e : t 1 y } p } e ` s p e c i f i e d b y t h e c o r r e s p o n d i n g X M L ' s t a g . T h e J S O N M U S T c o n f o r m t o t h e X M L f o r m a t , i n c l u d i n g a n y t y p e s a n d f o r m a t r e q u e s t s e . g . r e q u e s t s f o r l i s t s , o b j e c t s a n d s p e c i f i c t y p e s . B e c o r r e c t a n d c o n c i s e .

Followed by another call with this prompt:

I " H G O { e i o N w \ l v u o L a n p e t u Y s n p < < t m u s s p r g e b t t t u e i e > r r t t v \ c l i i > u e " o o n n r n p r w g g n e r t t e i n n a h _ c s a a e t t m m v y X e e a f p t M = = l o e h L " " i l \ i p n d l " s t e a o : h t m J w b a _ e S i \ y t t " O n " y N g d m d p d o a e e e o r g k s " s b e \ i c c j s " n r d r e p , g i e i c o \ b s p t n n i e c t s t s r i ( e i o n , v t p n o a h t = w \ l e i " o h " i o a t i n d i n h c a n = u e h m J f " n r e S o S i w \ O r p q t a " N m e u e s : . a c e x t i t n \ i e p o " o s e i t B n t s u o p d t f n n a d o a e r y p m c s " e e e e e x t " s a t " s b r a l a r e c y t ) a , s f r w J o h S m e O r N t e . h i t s h e d o k c e u y m e o n f t t a h n e d f t i h e e l d t a i g n s J t S o O N e x i t s r a t c h t e i ` t n a i m n e t ` o . a t t r i b u t e o f t h e c o r r e s p o n d i n g X M L , a n d t h e v a l u e i s o f t h e t y p e s p e c i f i e d b y t h e c o r r e s p o n d i n g X M L ' s t a g . T h e J S O N M U S T c o n f o r m t o t h e X M L f o r m a t , i n c l u d i n g a n y t y p e s a n d f o r m a t r e q u e s t s e . g . r e q u e s t s f o r l i s t s , o b j e c t s a n d s p e c i f i c t y p e s . B e c o r r e c t a n d c o n c i s e . I f y o u a r e u n s u r e a n y w h e r e , e n t e r ` n u l l ` .

Woof. That’s a whole lot of ceremony to get structured output!

source

Comment? Reply via Email, Bluesky or Twitter.