The yellow leaves could be other than lack/low fertilizer, among them lower light indoors than outdoors, also a root problem (too wet or too dry).

The ants are there because you most likely have sacle. To find them, look closely at the underside of the leaves and along the midrib. They can be quite small are often a very pale tan color.

And yes, the ants may be living in the soil. But if they are traveling up and down the stem, there's something up there for them. Most likely scale.

As far as not growing, citrus doesn't grow year-round but instead grows in several spurts.

As for which fertilizer is best -- they all work. Choose one and follow the directions for a potted plant.

As far as temperatures, all you need do is protect against frost. I kept my potted Meyer lemon outdoors and brought it in ONLY when frost threatened. Thus, its longest indoor stay was 5 days, then it went outdoors immediately.

Oh yes, you wrote: "flowered once, but then the flowers died and no fruit came from it." This is very common. You'll be lucky if 40 percent of the flowers set fruit. And potted plants are likely to set even less simply because life in a pot is full of risks.

I am a relative newbie at growing citrus myself, so take my comments with a large grain of salt. But, they are based on what I have either experienced myself or read about, mostly on this forum. As someone, who grows citrus trees in pots and moves them indoors for the winter, I thought Id share what little I know. Some of this overlaps what jean001 wrote, but I was on a roll, so just kept typing.

1) Leaf color change and leaf drop.
Change of leaf color and dropping leaves could be the result of:
a) Lack of key nutrients. Most people use some form of fertilizer for acid-loving plants, like Miracid, or, if you are lucky enough to find it, one specific for citrus. Also, many people supplement with foliar sprays that contain micronutrients, like Superthrive, fish emulsion, Spray-N-Grow, etc. Don't fertilize too frequently and many people dilute the fertilizer relative to the label instructions. Many also use a time release fertilizer, like Osmocote, but at a lower concentration than on the label. Potted plants tend to lose fertilizer more quickly, since it washes out, but they can also accumulate residue salts in the pot, which can be seen as a white crust on the surface of the soil. Potted citrus benefit from routine flushing of the pots with lots of clean water. In winter indoors they need even less fertilizer and many stop fertilizing completely until the trees go back outside. It depends on whether or not you see any growth that would justify fertilizing.
b) Too much fertilizer. You can over fertilize citrus, but I don't know how that is evidenced by the plant, since I try to stay on the lean side with fertilizer. I suspect you would see the leaves brown or Âburn before they dropped.
c) Pot size: If citrus trees are too small for the pot, then the roots can rot, because the soil around the roots stays wet, even if it appears to be dry at the edge. General sizing recommendations I have read are: one-year old plants: 8 -12 inch pot, two year old plants: 10 Â 14 inch pot. It depends more on the condition of the plants than on their specific age, since some grow faster than others and different nurseries provide trees of different sizes for the same reported age. Personally, I like my citrus trees (all true dwarf) to get a little root-bound, since I think it minimizes the risk of root rot, although you have to be careful to water more frequently. I pull slowly and steadily on the trunk, and if most of the soil in the pot does not lift out with the roots, I do not transplant that tree into a larger pot. Probably stunts the growth of my trees a little, but thatÂs okay, since they have to be moved in and out of the house.
d) Change of location: It is drier with less light inside and generally a more stressful environment for all plants, but especially for those that have been kept outside all summer. Some react by dropping leaves. If this happens, you should see new leaves and/or stems sprouting from just about every old leaf junction within a month. Some people mist their indoor citrus or keep them on trays of pebbles to raise humidity. Personally, I donÂt, more because I am lazy than for any other reason, and my trees did not mind last winter. You will definitely need to water less than when they are outside. If the plants are not getting enough direct sun each day (and they need several hours), you may need to supplement with artificial lighting.

2) Ants
Ants are usually a sign of scale, since they Âgrow the scale on plants to harvest the sticky sweet Âhoneydew the scale releases. Scale is usually on the undersides of leaves and can be hard to spot initially, since it will be small and green, eventually hardening into larger, brownish ÂscalesÂ, hence the name. Eradication of scale can be done by washing with soapy water, spraying hard with jets of water, using a Q-tip dipped in rubbing alcohol, scraping with your fingernail, spraying with horticultural oil, which is usually Neem oil, or just giving up and spraying with a pyrethrin-based insecticide, which is non-systemic and does not leave a residue. Watch for re-infestation, since it is hard to get them all the first few times.
If you have ants living in the soil of the pot, they will probably keep re-infecting the tree. In my opinion, you need to eliminate the ants completely; do not co-habit with them. Either use a pyrethrin-based soil drench and afterward rinse the soil really well, or else put an ant bait station on the soil and let them bring it back into the colony to kill it. Most bait station contents are rated as non-toxic to mammals, but you need to consider that some will be brought into the soil, so rinse it well after the ants are gone. Since you donÂt have any fruit and are unlikely to have any set in the near term, you donÂt need to worry about residues in the fruit.

3) No growth
Citrus trees go through stages of growth and rest. And, they may be recovering from the transition from outside to inside. If they are healthy, they will eventually start sending out new growth. Be sure not to overwater during this time, since, with few leaves, they are not transpiring much water and they can easily get root rot.

4) Dropped fruit
Most fruit is not kept by the tree to mature, because it would overstress the tree. This is especially true for young trees and even more so for citrus in containers. The recommendations I have read are not to even let citrus set fruit if less than two-years old, so more energy can go into developing strong roots and lots of canopy growth. You are probably just seeing the normal response of a young tree.

Hope all of this helps a little.

narcnh