This is absolutely something to worry about. Does your cat also eat dirt? If so, your cat may be exhibiting that it has an iron deficiency. It may be from anemia, which can stem from the bite of the wrong flea or tick that carries the disease. What will happen is that the cat's blood will start to become watery, and your cat will lose a lot of weight. Become lethargic, sleep a lot, and at the end stages, have a hard time keeping its balance and walking. A blood transfusion can help if caught in time. I lost my first cat from this exact thing. Take it to the vet.